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Preface 


For the thirty-seventh time, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the annual AECT Convention in Indianapolis, IN. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volumes 1 
and 2 are available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes 
are available to members at AECT.ORG. Proceedings copies are also available at: 

http://www.tresvstems.com/proceedings/ 

The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 


Michael R. Simonson 
Editor 
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In both research and teaching contexts, culture is often viewed in terms of ethnic and national diversity. 
While important on its own, such an approach neglects other crucial factors that along with their upbringing impact 
their acculturation into a group, namely, views and perceptions of self and others, educational and professional 
background, and interests (Hewling, 2006; Scollon & Wong-Scollong, 2001). Yet, these vary constituents are the 
one that allow or prevent acculturation into an environment. 

Culture is an integrated system of knowledge, beliefs, behaviors, customs, and experiences created and 
shared by members of a specific group (Gunawardena, 2003). Thus, having a particular heritage into which group 
members were born is no longer a singular aspect of what they are. Individual group members are stakeholders. 

They have the power to actively create, moderate, or simply adopt cultural norms of a specific environment, whether 
it is a community, a workplace, or an online course. Group members bring their individual characteristics, 
knowledge, and skills that impact interactions and collaboration among peers and with the instructor; language or 
professional jargon used, all in the framework of the class environment specified in the curriculum and scaffolded 
by the pedagogy and technology used (Goodfellow & Lamy, 2009). The clash between their own existing 
knowledge and skills and the acceptance of knowledge and skills possessed by other group members, as well as 
overall adoption of and adaptation to the forming culture requires constant re-negotiation of roles, styles, and 
expectations from all group members (Goodfellow & Lamy, 2009), which, in turn, is mediated by perceived social 
and knowledge capacity, boundaries of self-regulated learning skills, as well as interpersonal skills (Oliveira, 

Tinoca, & Pereira, 2011). For example, an atmosphere in an online class for instructional designers would differ 
from one for engineers not only in the technical content, but the culture of learning and communication as well. 
Should these two groups come together, they would create their own unique culture that would be viable for that 
new class. Flowever, once students leave the class for a period of time or forever, they would leave most of that 
culture behind, while possibly maintaining only some or even a version of some of those behaviors used in that 
unique culture. 

Y et, challenges may emerge when group members do not want to engage or do not possess the skills and 
knowledge to be an active and involved participant. As such, the pedagogy and scaffolding used can help structure 
the acculturation process and ongoing involvement in an online course, while supporting the individuality of group 
members. 

In this paper, we will investigate psychosocial characteristics of learners in an online learning environment, 
their perception of self and others, interaction patterns, as well as the effectiveness of strategies and supports that 
were offered to learners to scaffold participation and learning. 

Context: Course Redesign 

An online graduate program in Learning Design and Technology (LDT) program was launched in 201 1 at a 
Mid- Western University. The program is fast-paced, where students complete two eight week classes each semester. 
Courses from the online program are also available for traditional face-to-face students. It targets professionals 
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interested in the instructional design field from across the United States, who come with their own experiences, but 
do not necessarily have a background in learning design and strategies. As a result, it is salient for these learners to 
gain deep understanding of complex topics and evidence-based practices and apply their knowledge within a short 
time frame. 

Introduction to e-learning class is one of the required courses, usually taken closer to the end of the LDT 
program. It focuses on the development of instructional design skills for online course development. During the 
course, students need to develop an e-learning module based on the topic of their choice. Throughout the eight 
weeks, students participate in a number of discussion activities based on weekly readings; complete three project 
phases (proposal, paper prototype, and final e-learning module), and also submit their final reflection or self- 
assessment on the process, what they have learned and how they generalized knowledge from previous classes and 
applied to this project and process. Students are also asked to provide peer feedback on three items: objectives, 
assessment piece, and paper prototype. 

While students liked the course as gathered from course evaluations and from speaking with individual 
students and course instructors, many students experienced a number of challenges: 

Students could use an instructional design (ID) model of their choice. However, the fast pace of the class 
and an ongoing iterative process did not align well with some ID models used, resulting in confusion and 
frustration. 

Time management skills were also a frequent challenge for the learners. Often, this was rooted in the fact 
that students did not realize the scope of the project or the time needed to get comfortable with the 
authoring tool or learning management system, thus, leaving a large portion of project development until 
the end of the course. 

They felt that they were on their own with the design, because the three feedback opportunities offered 
during the course were not sufficient to receive peer feedback to help them with creating a strong e-learning 
module. 

Students felt course discussions based on readings tended to be more general in focus and did not 
necessarily support project development phases. 

Finally, students were at different levels of readiness to design and create an e-learning module, therefore 
many students felt that their individual needs were not met. 

To scaffold their design and development process and also help students create a community with a 
supportive learning culture, the course was modified and piloted in Summer 2015. The following changes were 
introduced: 

Use of rapid prototyping as an instructional design model to acquaint students with an ongoing revision and 
feedback process while they work on individual components of their project. 

A compilation of reading materials on general e-learning development principles, project management, 
peer critique, and self-regulation skills for an e-learning field (e.g., goal setting, planning, time 
management). Such readings included both peer reviewed articles and professional publications. The 
selection of professional publications allowed students to see best practices in action. The selection of 
topics for reading was based on experience with prior students and frequently voiced out concerns, as well 
as unofficial observations in the field and personal experience. 

Weekly peer feedback opportunities for each step of project development. In addition to improving 
students’ project, the goal of peer feedback opportunities was to increase prosocial behaviors and 
communication among students to hone their skills, and create multiple opportunities for students to start 
discussions on topics that are personally meaningful, learn from each other, and thus create a stronger bond 
that would unite students within the classroom. After all, when in the field, instructional designers need to 
effectively communicate their designs at any stage of its inception and development. 

Methods 


Participants 

Forty three graduate students enrolled across four sections of the Summer 2015 eight-week course that 
introduces students to principles and practices of developing an eLearning project. From them, 28 students (pre-test, 
included demographic questions) and 33 (post-test). Two students are from the traditional on-campus Master’s 
program. 


200 



Based on the pre-test, the majority of students (82.14%) were working towards their first Master’s degree. 
The majority of students were females (78.57%). The ethnic composition was as follows: African American (1 
person); Latino/Hispanic (2); White (20); other/mixed (2); and preferred not to answer (1). 

Students’ professional background was as follows: 

• Educators: 57.14% (39.29% from K-12) 

• Business sector: 25.0% 

• Unemployed: 7.14% 

• Other: 10.71% 

Finally, 60.71% of students reported that they had no prior e-learning experience, while 28.57% had limited 
e-learning experience through their workplace. 

Instruments and Procedures 

Interviews: Individual student and faculty semi-structured interviews were conducted over the phone after 
the end of the semester. The use of interviews allowed the researchers to gain a more robust understanding of 
perceptions about the class, progress made, and challenges experiences. Participants could also provide additional 
information or experiences they felt were important but may not have been specifically requested by the interviewer. 

Surveys: Online surveys were administered in the beginning and end of the courses. Survey instalments 
included Likert-scale and open-ended questions to clarify and provide in-depth opinions about a range of topics 
covered in the course, acculturation processes, as well as self-reported level of self-efficacy, self-regulated 
behaviors, self-motivation, sense of community and the culture within, as well as experience with feedback were 
assessed both at the beginning and at the end of the course. Due the scope of the larger research study, the survey 
instalment included questions aimed to investigate a range of opinions and skills. 

Artifacts: Student artifacts, namely the end-of-class final assignment reflection paper as well as reflections 
shared on the discussion board were reviewed to gain a contextual understanding of student experiences. 

Data Analysis 

Descriptive statistics were used to summarize closed-ended survey data. In addition, qualitative data from 
interviews with students and instaictors and open-ended survey items were reviewed using a descriptive coding 
approach to discover topics or themes related to the use of badges in students’ learning experience. The themes were 
then analyzed and grouped under categories (Saldana, 2009). 

Findings 

The analysis of the artifacts, surveys, and interviews revealed the following three major categories that may 
impact student acculturation: 

Establishing self: self-identity and group membership 

Building relationships: peer-to-peer and peer-to-instaictor interactions and collaboration 

Course environment: course features, staicture, and scaffolding. 

Establishing self: self-identity and group membership: 

Acculturation into a new environment creates the basis for ongoing negotiation of learner’s self-identity 
and roles (something that they already possess), as well as group membership both within the program (e.g., cohort 
system) and outside (e.g., employment) against others in attempt to create the learning culture of a new course. As 
such, they try to project that self-identity to impact opinions of others about themselves and how they do in 
comparison with peers. All in all, they engage in role negotiation within the emerging learning culture and 
community, in particular. Survey responses showed that 63.6% consider discussions/feedback as a way to establish 
their expertise within the group. 

I would respectfully disagree ...Here's a sampling of some different e-learning courses I have worked 

on... 

Some of my peers in this course seemed to lack quality and experience in instructional design. Therefore, 

it was hard to "rely" on their feedback or input. 

I want be that student that people can look up to and ask questions if they have questions or whatever. 

I was intimidated by classmates who appeared to be much more knowledgeable than I was. 

Additionally, learners seemed to be more proactive in their collaboration efforts with their “in-group” 

peers. 
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You have your same people that you've been in other classes with and you kind of develop a rapport. 

Those seem to be the ones you get the greater response from. 

The inability to claim group membership often created the feeling of unease, tension, and signs of low self- 
confidence. 

A student who had to miss a semester for personal reasons, started this class with a different cohort and 
initially struggled while getting to know them. She was successful at the end, but claiming a new group 
membership was a challenge at first: That was a little difficult at first. They don 't know me. They're not 
used to me yet and now I'm getting to know them. 

Because I'm not working outside of the house right now. I work in my house. So I don 't have the same 
experience as these other people do. I'm notan instructional designer. I'm going to school to be one, but a 
lot of the people in class are instructional designers. 

My challenge with providing peer feedback, is that I worry’ about writing the wrong thing, or something 
that they will not agree with. I worry’ that sometimes if I offer the feedback 1 might hurt their feelings. 

Others may be quite open to new experiences and ideas and learn from each other. However, one may 
wonder whether it were a true excitement or a way to balance the unease with a more positive outlook. 

Because everyone comes from various backgrounds, with the discussion boards, you get to share that. You 
get to share everyone’s experience with this one [discussion - auth.] question. 

Building relationships: peer-to-peer and peer-to-instructor interactions and collaboration: 

In the process of acculturation, active interactions with peers, particularly, who are not in their current “in- 
group,” helped with adaptation in the class by reducing the feeling of inadequacy or being lost, as well as ongoing 
encouragement to keep up and stay on track. Based on the survey response at the end of the course, 81.8% felt that 
online discussions helped them stay connected with peers. 

I think the feedback really helped with that [building a community - auth.]. Being able to provide feedback 
and actually building relationships that way. 

The interaction with classmates through discussion boards was vital to reaching my project goals. Some of 
the interactions continued through email exchanges where my classmates offered support, feedback and 
even a word of encouragement. 

Some students start forming new relationships based on misgivings, like struggling with the same topic. 
Another part of their feedback that became very significant to me in this process was their honesty with 
the problems they were experiencing as well. 

I had a few people we commiserated all of our experiences so we bit off a little bit more than we could 
chew from the start... It was like, oh, thank God, you too, not just me. So I think it helps the way that the 
sessions were structured, it really helps to get to know people and to build those relationships. 

Learners tend to be more comfortable and less intimidated by learning from peers creating the supportive 
network and creating avenues to learn from each other. 

For me, it's easier to ask my peers before I ask my teacher. I don 't mind if my peers might think it's a 
stupid question, but it would bother me more if my professor thought it was a stupid question. 

Y et, some preferred creating new closed “in-groups” or staying with the existing ones from previous 
classes, separating themselves from the rest of the peers: 

So one thing about this class that I really liked is that we were allowed to partner. And so my partner and I 
have had several classes together, so it was a really good fit for both of us. My partner and I are very 
experienced instructional designers. 

Instructor’s presence and support did not seem to dominate their perception of the class. Students tended to 
appreciate help when they asked for it and for encouraging reaching out to other students. However, for students 
who perceived themselves as the ones not having peers, an instructor would become the main contribute into the 
project design. 

[M]y instructor was a valuable resource for me during this project. When I felt as if I had gone astray or 
had lost my focus, I would reach out for feedback. 

I've found that my willingness to reach out depends on the instructor and how welcoming they make the 
class feel. 
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My partner and I are very experienced instructional designers, so honestly, it was [instructor] 's feedback 
that was the most important to us. 

Asynchronous class and inability to communicate face-to-face may have inhibited relationship building: 

I tend to do my work during the week and most people tend to post on the weekend. It makes it hard to 
stay in the conversation. 

[...] written feedback is never as detailed or creates as much of an impact as face-to-face feedback. 
Oftentimes, more can be communicated in a five-minute meaningful conversation with another than in a 
multiple-paragraph discussion post 

Course environment: course features, structure, and scaffolding 

The additional tools and strategies that were added to the course to support their project development and 
further acculturate into their program, as well as a career of choice were strongly appreciated by the majority of 
students. 

Peer feedback process was found helpful by 95% students: 

Peer feedback has been the most valuable part of this course. With peer feedback at almost every step of 
our development and as we worked through the prototype we were allowed a second set of eyes that are 
capable of viewing things in the context of our actual assignment. 

Resources on giving and receiving critique were found helpful by 88% students: 

I think that the articles, the resources we were given, that reminded us to what we’re really giving the 
feedback for. I paid attention to that. That made me go back and try a little harder. . . I think I naturally 
understood that I should be very honest, but having that guideline, it was much easier to say, "Okay, now 
let me go a little deeper with it. " Had my peer not read the same information, I would've felt a little 
awkward. 

While students may not have been as happy about added readings, tools and resources on self-efficacy, 
motivation, and self-regulation were found helpful by all the students, even though some said that it was a review for 
them. However, time management was still one of the challenges most frequently listed in self-reflections 

Some of the self-management activities presented in the course helped me become more organized and in 
control of my time and project goals. I set smaller, more attainable goals as well as longer term goals. I 
also made to-do lists less overwhelming by prioritizing project tasks. 

Even more so, learners started developing the sense of responsibility not only for their own work, but also 
for the work of others and made proactive attempts to support their peers’ learning. 

Because of the iterative process [of the newly redesigned class - auth.], I found myself often, before giving 
feedback to someone, going and looking at the previous discussions over the past few weeks to see what 
they cut out, what feedback they received to try> to see what iterations they were making so I wasn 't just 
giving them circular feedback. 

However, lack of understanding of professional jargon, essential technology, or similar components of a 
course that could impact their productivity may also create the distance between an individual learner and the 
learning community, if a learner feels that he or she cannot operate on the same level or efficiency. 

Even just some of the vocabulary’, because I started to realize that some of the vocabulary used in my 
program, just the comments people were making, is vocabulary’ used in all authoring programs. Because 
it's sometimes hard to get help from the normal feedback because you don 't know how to ask the 
questions. I mean, you know what you want to do, but you don't know the vocabulary’ and you can't find it. 

Perception of the learning community as part of acculturation 

While the survey results showed that students developed trust to their peers in comparison with a 
previously taken class (p<0.001), when students were directly asked as to whether a learning community was 
formed, perceptions were often negative and somewhat strongly worded. 

Some of us are just trying to get through the program and learn what we need to learn and be done with 
the program. 

I don 't think these classes feel like a "family". And for me, I don 7 think they ever will. People are too busy 
with their personal lives, schoolwork, actual work, etc. 

I feel absolutely no connection to the course, the professor, or the other students. When I'm forced to 
engage in "get to know you" activities at the beginning of a course, or required to post a certain number of 
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times on a discussion board, I honestly do the bare minimum, just to get it done. [36.4% agreed on this - 
auth.] 

On the other hand, others felt more positive about their experience: 

I think that there is maybe just an inherent sense of community just because we're all there for the same 
purpose. 

I have enjoyed being part of an online community that is comprised of individuals with such diverse 
professional backgrounds. I feel I have learned a lot from communicating with these different people. 

The discussion boards also gave the atmosphere of camaraderie... 

However, considering the terms they used to describe a community, there was not much agreement as to 
what a community means. 

Discussion 

The perception of learning culture cannot be considered without the taking into account student self- 
identity and background that they bring into the course culture as well as the negotiated practices of interpersonal 
communication along with the structure imposed by the instructor (e.g., pedagogy; Goodfellow & Lamy, 2009). 

The findings showed that even though students were struggling with some components of negotiation in 
terms of their role or place in this new learning culture, the overall attitude was rather promising. Students tended to 
proactively look for ways to not only gain something from others (e.g., helpful feedback), but took proactive steps to 
provide even greater support to peers than expected, which shows the prosocial behavior of a healthy learning 
culture. 

However, such prosocial behavior required an infrastructure (i.e. course structure, means of collaboration, 
etc.), at least in the beginning, to initiate the conversations and help learners find their place. 

Yet, in spite of these prosocial behaviors, when asked directly as to whether they feel part of the course and 
the learning community formed within the course, the overall perception was quite negative. In part it can be 
explained by individual perception and expectations of what an online learning culture or sense of community mean. 
These notions are much more nebulous than the idea behind more specific terms, like feedback or technology 
limitations. Even more so, such experiences differ significantly from a face-to-face classroom where students can 
get to know each other more efficiently in and outside the classroom. As such, in comparison, learners may tend to 
downplay their online experience as lacking in comparison. 

These attitudes can also be explained by the more traditional direct teaching approach where student is a 
recipient of knowledge with a pre -designed structure, environment, and a community. As such, they may not feel the 
need to make an effort to build the community or proactively engage in a class beyond the syllabus requirements. 

Implications 

Based on the course findings, the following implications and considerations for future course redesign can 
be outlined: 

1 . Since learner acculturation is connected with their self-identity and rooted in individual expectations, some 
students distanced themselves from others based on their perception of self and sense of belonging. As 
such, the following considerations need to be made: 

From the beginning of the semester, purposefully allow students to 

• Co-construct their notion of a community, 

• Determine how their individual strengths can be helpful to others, 

• Identify academic and non-academic skills or competencies they may want to develop 

• Determine if those skills are already possessed by other students. 

2. Regular peer critique helped students with skill development, project improvement, and creating avenues 
for meaningful interaction and collaboration. However, lack of professional ID experience impeded some 
students from providing meaningful peer feedback. As such, potential modifications include scaffolding for 
less experienced students, e.g., suggestions for a structured review sheet/checklist to overcome the fear of 
providing critique to their peers. 

3. Project management and self-regulation readings helped students start shaping their own strategies that they 
tended to share with others while building their self-confidence. However, this added to the heavy reading 
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load. Even more so, not all students were able to successfully apply these strategies to their projects. 
Therefore, reducing the number of readings and include more activities that help develop and utilize 
strategies in different contexts to help with skill generalization. 

Limitations and Future Directions 

The study has some limitations, including a small sample and time limitations of the course period that may make it 

insufficient to develop stronger skills or a sense of community. We plan a new phase of redesign to be implemented 

in Summer 2016 that will incorporate the current findings to improve student acculturation process. 
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and Implementation of an Educational 
Competitive-Trivia Mobile Application. 


Sean D. Bailey 
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Descriptors: Mobile Game Design, Classroom Media/Gaming 

Overview 

This study details the stages of researching, designing, and developing an educational, 
competition-driven mobile application for middle and high school students. Everything from idea 
conception to design strategies to integration of research results are reviewed. This research includes topics 
such as lessons-learned, development issues, usability testing, and “Aha!” moments. Discussion of the 
possible obstacles and successes that similar designs and implementations could introduce occurs. 

TriviaPrep 

An ever-growing trend in education is the integration of mobile devices in the school setting, 
especially considering the abundance of research on “BYOD”. Not surprisingly, a common concern found 
is the lack of student interest in educational applications. Thus, these concerns have fueled the passion 
behind this design development of this tool. 

With trivia/quiz games like “Trivia Crack” and “Quiz Up” littering the best selling charts of the 
top mobile applications, one can’t argue that competitive trivia gaming is ‘all the buzz’. An application 
designed to harness the competitive and addictive qualities that these offer while replacing the random 
trivia questions with core curriculum concepts could be a major asset to the learning culture in the school. 
TriviaPrep is an idea that has been designed with that possibility as its emphasis. 

“You know, something like this could change the entire culture and atmosphere of the school. ” 

- Paul Prater, Principal, Bath County High School, KY. 

It can be said that competition breeds excellence. When competition is integrated into a learner’s 
environment it fuels focus on their individual success as they make comparisons to their peers (Lam, Yim, 
Law, & Cheung, 2001). A driving factor behind the design of this application is the integration of school- 
based leaderboards. As users play this multiple-choice quiz game, they can compare their scores with that 
of other students in their grade level and across the school. Recognition of “Top Students” in each class is 
essential and will be highlighted throughout the multi-platform application. Weekly and monthly 
leaderboard reset options are currently being evaluated to retain competitive value. 

The process of evaluating one’s own abilities derives from comparison to others. Naturally, 
students are creatures of comparison. In order to determine our level of success, or lack thereof, we 
compare ours with the achievements of those surrounding us (Gilbert, Giesler, & Morris, 1995). 
Competition, therefore, results in motivation to improve. This application is designed around this core idea. 

So, what exactly is the content of the trivia questions and why? 

The answer to that question is this: Common annual high school effectiveness measurements heavily favor 
the results of annual standardized testing and/or undergraduate admissions tests (ACT, SAT, etc.). The 
questions asked will be collected from national standardized assessment practice questions and will be 
categorized accordingly. Essentially, students will be working toward improving assessment scores while 
competing with classmates across their school. 

Another key design goal to integrate fun into the application and to mask the applications 
appearance as “just another learning app” is the blending of randomized sports, entertainment, and “School 
Spirit” trivia items into the game . These items occur at a 20-30% likelihood. The “School Spirit” items 
will be user/school submitted questions covering random facts about their school and community. 
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To remedy possible educator integration apprehension, development of teacher and administrative 
resources for numerous school and/or district wide integrations of the application will be developed before 
release. The website www.triviaprep.com will be used for educator resources and forums. 

With a target of a school-wide pilot implementation in the coming months, the development of the 
mobile application will be broken into numerous phases that will later be determined by further study . 
Currently the application is in the final stages of research, development, and design. 

Design and Development Overview 

• Application name: TriviaPrep 

• Initial target audience: High school students, ages 14-18. 

• Initial trivia content: ACT and SAT practice questions, blended with “random fun” 
questions and/or “School Spirit” trivia. 

• Scoring: Correct Answers, Speed (based on question type), Multipliers (Consecutive 
correct answers, wagers, “Rock/Paper/Scissors”) 

• Logo: 

Trivia Pre^ 

• Splash Screen mockup: 



• Navigation/Home Screen mockup: 

TriviaPrep 



• Website: www.triviaprep.com 

• Programming language: LiveCode 

• Platforms: Multiple (IOS, Android, and HTML5) 

• Initial research and school-wide integration pilot location: Bath County High School, 
Owingsville, KY 
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The quality of research that goes into design and development prior to release may contribute 
significantly to the success of an application. Informal discussions, student focus groups, educator surveys 
and testing results are to be integrated into the research. 

Closing: 

The development of this application is approaching its final stages. The processes, obstacles, successes, 
and frustrations that occur during the development of an educational application from idea to product is 
emphasized. This research may serve as a valuable resource for those interested in mobile application 
design and development. 
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Abstract 

This paper shares the initial results of an ongoing design-based study that uses social network analysis to 
gather feedback during the development of a campus-wide implementation of the Quality Matters™ (QM) standards 
for online courses. A review of the literature on the use of design-based research and social network analysis for 
program development is included. The authors share the design, analysis and findings and how the iterative analysis 
will inform changes in subsequent development cycles. 

Introduction 

This study was designed to provide insight into a problem that needs to be addressed and resolved for all 
institutions of higher education. The problem is how to develop standards for online course quality that apply to all 
courses in a collaborative manner and without intruding into the content of the course. 

Our institution chose to use Quality Matters™ (QM) Standards (Quality Matters, 2015) as the standard for 
quality in online courses at our institution, which is a regional four-year university in the northern Midwest. We 
initially designed a basic institutional (internal) standard set, which was aligned with the Quality Matters™ (QM) 
national standard. The internal standard was designed in such a manner that courses that met our internal quality 
standard, with minimal additional effort, could be enhanced to align with the Quality Matters™ National Standards. 

We choose to conduct a designed-based social network analysis as we implemented the development 
activities with faculty and staff. The QM team agreed that the Phase 1 network would be defined as the time period 
between the first offering of the initial Application of the Quality Matters Rubric (APPQMR) Training on 
11/21/2014, up to the (but not including) an APPQMR session on 09/18/2015. As a result, 09/18/2015 becomes the 
initial date for Phase 2 of the NIU QM network. The end date of Phase 2 has yet to be determined, but will likely 
continue to parallel the academic year. This article reports the results of the analysis and the outcomes of Phase 1 of 
the implementation of QM Program. 

This report indicates what the analysis shows and shares considerations for future program development 
based on the Phase 1 analysis. It should be noted that, as with all social network analysis, this analysis, in and of 
itself, is not intended to reflect the entire reality and complexity of the QM Program development and operation. Our 
intention is to use this data and analysis, combined with our lived experience of the process, as a starting point for a 
conversation about our program development to date and what to do to enhance the quality and success of the 
program moving forward. 


Quality Matters™ (QM) 

The roots of the Quality Matters™organization can be traced back to MarylandOnline, Inc. and a three-year 
Fund for the Improvement of Postsecondary Education (FIPSE) Grant to develop a quality assurance standard for 
online courses that could be used for inter-institutional quality assurance (Office of Post-Secondary Education, 
2015). MarylandOnline, Inc. eventually transitioned into the Quality Matters™ group. This group provides a 
standards rubric that consists of 43 specific, research-based, and effective online course design elements. The QM 
Program includes processes for rigorous internal and external peer-reviews and ultimately offers national 
recognition to those online courses assessed as meeting the required standards in reviews by expert reviewers. The 
program also includes extensive faculty and reviewer development programming. The initial training in QM is the 
Application of the Quality Matters Rubric (APPQMR) training. This is a 1-day session where participants are 
introduced the the Quality Matters Rubric and spend time applying the rubric to samples of online courses. Our 
survey was taken by APPQMR participants at the end of the day-long APPQMR workshop. 
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Design-Based Research 


Once standards were established, the next step in the process was to design the implementation of the 
standard. It was at this time that the research commenced. Members of campus eLearning, Faculty Development and 
Outreach and Engagement offices are conducting the QM Program implementation and and research. It was logical 
to pursue a design-based research approach because the study presented the need to iteratively design and develop a 
program. Design-based research is identified as a research method well suited to environments that are facing rapid 
changes, such as the rapid emergence of new technologies and processes in education. Wang and Hannifin (2005) 
define design-based research as: 

A systematic but flexible methodology aimed to improve educational practices through 
iterative analysis, design, development, and implementation, based on collaboration among 
researchers and practitioners in real-world settings, and leading to contextually-sensitive 
design principles and theories (p. 7). 

They go on to describe design-based research as consisting of 5 main components. Design-based research 
is: (a) pragmatic; (b) grounded; (c) interactive, iterative, and flexible; (d) integrative; and (e) contextual. Design- 
based research (DBR) involves the concurrent application of design, research and practice. Design-based research is 
one of a variety of approaches that have emerged and/or gained popularity in response to rapidly changing 
conditions, diverse and complex technical or social dynamics and the significant needs of areas ranging from 
technology hardware and software development to public school reform. Examples of these models include design 
based research (Wang and Hannefm, 2005), action research (Lewin, 1948), practitioner research (Anderson, Herr 
and Nihlen, 1994) and experiential learning (Kolb, 1985). These models all share a common process that is cyclic 
and moves through phases of problem identification, question formation/research focus, intervention and analysis. 

At the conclusion of the analysis, the cycle begins again - drawing upon what was learned in the previous cycle to 
improve the next iteration of the cycle. In addition, these processes involve direct, active and collaborative 
participation with stakeholders in the research environment. 

The power of this research model is its capacity to bridge between theory and practice to generate solutions 
in both a timely and effective manner. Design-based research is driven by a disposition toward connecting design 
interventions with existing theory (Barab and Squire, 2004). In this case, the DBR process led to the applied use of 
surveys and social network analysis to concurrently inform the design and development processes. 

Social Network Analysis 

Social network statistics and visual renderings provide a powerful perspective on the activities in networks 
of various types and are used to develop strategies to enhance networks (i.e. improving communication in an 
organization) or inhibit networks (i.e. disrupting terrorist groups, stalling the spread of infectious diseases). The 
analysis of social networks can be traced to the early 1930s when Mereno (1934) introduced sociograms, which 
were hand drawn diagrams of dots representing members of a group and lines with arrows representing connections 
between group members. 

The more contemporary form of social network analysis evolved when the social sciences merged with 
mathematics in identifying patterns in network structures. This merging led to the capacity to identify the structure 
of social networks and the roles and power differentials of individuals in these networks (White, Boorman & 

Breiger, 1974). Since that time, social network analysis computer software applications such as UCINET (Analytic 
Technologies, 2010) were developed. These applications draw statistical data from matrices to render network maps. 
Social network statistics and visual renderings provide a powerful perspective on the activities in networks and 
provide insights into future actions to enhance or inhibit networks. 

The three basic elements of a social network analysis are nodes (people in human networks), connections 
(lines between individuals who are in some type of relationship or information exchange) 

and attributes (particular qualities of a node such as age, years on the job, etc.). These basic elements are assessed 
using a variety of statistical measures to determine the strength or weakness of a network. Our study of the initial 
network included gathering and analyzing data related the connections between participants in the program and 
exploring how the attributes of the participants might have impact on the development of the QM Program. 
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A Social Network Analysis of the Implementation Quality Matters 

The social network analysis survey from which the raw data was gathered included a series of attribute 

items: 

• Location - The participants’ employment location (College, office, unit); 

• Role - The participants’ job function (professor, chair, supportive professional staff); 

• Power Ranking - The combination of a participants’ rank, experience teaching and experience 
teaching online); 

• Interest/Capacity - The participants’ willingness, desire or capacity to support the QM Program 
implementation. 

The survey also included three relational items that required respondents to check the names from a roster 
list of names based on three different types of interactions: 

• With whom have you worked to develop online content? 

• With whom have you worked on QM? 

• From whom would you seek advice on a work-related issue. 

Survey data was collected between 1 1/21/2014 and 09/18/2015. The survey was administered to 54 
participants who chose from a list of 56 possible individuals for each relational questions. Only 2 of the 56 
participants did not complete the survey - yielding a 96% response rate. 

Cohesion in the Initial QM Network 

The initial analysis of the QM network examined the cohesion of the overall network. Cohesion refers to 
the degree to which actors (nodes) in a network are connected to each other (Hanneman and Riddle, 2005). Unlike 
other quantitative statistical measures, there are no set thresholds that indicate the presence or absence of an effect in 
social network analysis. Table 1 below lists the cohesion data from the composite initial network analysis that 
emerged based upon the three relational survey items (worked with, worked with on QM and advice seeking). The 
table also provides definitions for each item and a column with an indicator of whether future progress would result 
with an increase or decrease for each measure. 


Table 1 


Measure 

Definition 

Initial 

Positive Indicator 

Density 

The number of connections present as a percentage of the 
total possible connections. 

.136 

t 

Degree 

The number of connections per node. 

7.46 

t 

Components 

Portions of the network that are disconnected from each 
other. 

25 

l 

Fragmentation 

Percentage of the network that is disconnected 

.55 

i 

Table 1 Initial QM Network Cohesion Measures 
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Figure 1, Figure 2 and Figure 3 below show the visual representations of the three network components that 
emerged based upon the three relational survey items (worked with, worked with on QM and advice seeking). 

Figure 4 shows the composite network (the three network components combined). The main data source for the 
study was the composite network, but there was also value in examining the components that formed the composite 
network. 



Figure 1. The NIU QM SNA Phase 1 network - worked with to develop online content. 



Figure 2. The NIU QM SNA Phase 1 network - worked together on QM. 
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Figure 3. The NIU QM SNA Phase 3 - advice seeking. 



Figure 4. The NIU QM SNA Composite Network 


Cohesion Analysis 

What was learned from the initial cohesion analysis was either significant or provided an important basis 
for future analysis when this Phase 0 1 network becomes the basis for comparison to the Phase 02 network. 

The statistics from the initial cohesion analysis indicated that: 

1. There were very few isolates (individuals with no connections to others) in the component networks. 
This means that relationships already existed between participants entering the program and that a high 
number of connections existed in the institution between individuals who are stakeholders in the 
development of online coursework. 
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2. The connections represent a diverse mix of types - connections between faculty, between faculty and 
department based staff and connections between faculty and university offices (eLearning and Faculty 
Development and Outreach). 

3. Of the network components, the strongest network was the advice seeking network, followed closely by 
the online content collaboration network. By far the weakest network component was the component of 
connections based collaboration on Quality Matters - which would be expected because the program in 
an inception phase. 

Though a cohesion comparison with a future network analysis will yield much more data and possibilities for 
analysis, the initial cohesion analysis provided our group with important areas to consider as we move forward with 
program development: 

1. There is a high degree of connections to people in the group who are sought for advice. How those who 
are seen as advisors feel about QM is an important consideration. 

2. There are a number of collaborative relationships in existence based on the development of online 
courses. These peer-to-peer and faculty to support staff connections are a resource that can be used 
share information and gain stakeholder feedback. 

3. Each support office (eLearning, Faculty Development and Outreach) have made significant connections 
to stakeholders. This indicates that the more the support offices collaborate, the more reach each office 
will have and this collaboration is also necessary to assure that stakeholders are receiving the same 
messages regarding the QM Program. 

In order to specify further action and to move from an overall network cohesion perspective, to a study of 
individual dynamics in the network, we completed a centrality analysis of the Phase 0 1 QM network. 

Centrality in the Initial QM Network 

Centrality refers to how central a person is in a network as it relates to aspects such as their number of 
connections to others, distance from others in the network and the degree to which a person is located between 
others on pathways in a network (Hanneman and Riddle, 2005). The centrality analysis of the Phase 01 network was 
accomplished in our study by examining an overall centrality score that was derived by combining and several types 
of centrality, including (Hanneman and Riddle, 2005): 

• Degree Centrality - The number of connections coming into a node from other nodes (how many 
people selected an individual as someone they are connected to through communication or 
collaboration). 

• Bonacich Power Centrality - An iterative estimation approach that weights each node’s centrality 
by the centrality of the other nodes to which it is connected. Node centrality depends not only on 
how many connections it has — but also on how many connections its neighbors have (and on how 
many connections its neighbors’ neighbors have, Etc.) 

• Closeness Centrality - Power that comes from acting as a "reference point" by which other actors 
judge themselves, and by being a center of attention whose views are heard by larger numbers of 
actors. Actors who are more reachable by other actors at shorter path lengths have favored 
positions in a network 

• Betweeness Centrality - A measure of the number of times a node occurs on a path from one node 
to another node in a network. 

The first three of these measures (degree, Bonacich and betweeness) have both an “in” degree factor and an 
“out” degree factor. In defining connection in a network, three possibilities exist: 

• Possibility 1 : A can choose B 

• Possibility 2: B can choose A 

• Possibility 3 : A can choose B and B can choose A 

In instances where an individual identifies a connection to another, the connection is considered an 
outgoing connection. The connection is identified as incoming connection in a situation where an individual is 
chosen by another. Connection is considered reciprocal if each person chooses the other. The centrality data used in 
this study is based on the in degree - meaning that we examined the incoming connections. A person must have 
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been chosen by another person be considered for centrality in our study (therefore possibility 1 above was 
excluded). 

The resulting data (see appendix A) demonstrated that centrality was concentrated in members of the 
research leadership team (eLearning, Faculty Development and Outreach). This is not a surprising finding because a 
majority of session participants were members of the eLearning and Faculty Development Departments. However, 
this was not the sole reason for high centrality. These participants’ centrality was also high because these individuals 
work with clients and groups across the entire campus and make connections in ways that those who work in 
individual departments or colleges do not. The degree of centrality of those in the leadership team bodes well for 
program development. 

Attributes and Power in the QM Phase 1 Network 

The most robust and telling visualization of the Phase 1 network was revealed when attributes were added 
to the network dynamics. As stated previously, there were a number of key attributes collected via the survey. These 
attributes included: 

• Location - The participants’ employment location (College, office, unit); 

• Role - The participant’s job function (professor, chair, supportive professional staff); 

• Power Ranking - The combination of a participants’ rank, experience teaching and experience 
teaching online); 

• Interest/Capacity - The participants’ willingness, desire or capacity to support the QM Program 
implementation. 

Figure 5 below shows the QM Phase 01 network with attributes. Several features of the network were 
identified as potential indicators for action based upon this data as we move forward with the implementation of the 
program. These will be discussed in the analysis and conclusions section that follow. 



Number = Participant Role Shape = Participant Location Size = Power Ranking Color = Participant Interest in QM 

□interested and Able □ Need More Information □Not interested or Not Able 

Figure 5. The NIU QM SNA Composite Network with Attributes 
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Analysis and Conclusions 


The following conclusions were drawn from the analysis and supported by our experiences in 
implementing the QM Program: 

1. A number of people with high power rankings participated in the initial sessions. We take this as an 
informal indication that online courses and any issues that might have impact on online courses are 
viewed as a priority because high-power individuals took the time to attend a full-day workshop. 

2. There are a number of high-power individuals who need more information before they can make a 
firmer commitment to the QM Program. High-power people have influence and so it will be important 
to follow up with these individuals and make sure they receive whatever further information they need 
to make them comfortable supporting the program. 

3. The campus eLearning, Faculty Development and Outreach personnel who are implementing the 
program have high power and centrality. This is a positive indicator as it indicates that these 
individuals have trusting relationships with stakeholders in the community and this indicates that the 
program implementation should be a smoother and more positive process because of the trust 
established by these individuals in work prior the implementation of QM. 

4. There are a number of high centrality, high power individuals who are not willing or able to support 
the program at this time. Because these individuals have a high degree of influence it will be important 
to follow up with these individuals to determine their status and to see what can be done to support 
these individuals (and those connected to them) to enhance their desire or capacity to support and 
engage with the QM Program. 

Our analysis shows both positive indicators and challenges in the future implementation of the QM 
Program. The challenges we currently face include the fact that 35 of the 54 of respondents (65%) indicate that they 
either cannot support the program or need more information. While 18 of these (33%) indicate that they need more 
information, 17 of these individuals (31%) are not able to support the QM Program for other reasons that cannot be 
remedied simply by supplying further information. In narrative commentary, the two primary reasons stated for not 
engaging with QM included a lack of time and, for faculty, that there was no credit awarded in the tenure and 
promotion process for their working on their online course development. This shows that QM is an institution-level 
implementation that will require institutional-level responsiveness to issues such as these. Because the Phase 1 
implementation was likely attended by early adopters, who tend to be leaders and have more positive attitudes, our 
results are likely to skew toward the positive and we can likely expect more challenges to emerge with future cycles 
of implementation. 

The positive indicators coming out of the analysis are that we have 54 community stakeholders trained on 
the QM rubric and aware that there are standards for quality in online coursework. Whether participants indicated 
they could support the QM program or not, participants did express that they planned to attempt to apply the QM 
rubric to their existing courses. A second positive indicator is that there is a strong network of existing connections 
between community stakeholders and these connections have developed over time and involve trust that develops 
from a history of providing quality support. Finally, a third positive is that the team implementing the QM Program 
(eLearning, Faculty Development and Outreach) have bonded over the implementation. Attending the QM National 
Conference, making plans for the APPQMR sessions workshop and collaborating to establish standards for our 
institution led us to leave our siloes and come together over important issues, to collaborate on the resolution of 
these issues and to collaborate on the implementation of the program. This has created a unified effort that benefits 
us as individual departments, as the QM Implementation Team and is particularly beneficial to our clients - the 
university community - who can expect consistent training and course development support. 
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Appendix A 
Centrality Data 
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42 

34 

6212.858 

106 

72.222 

2 

8 

30 

5969.041 

114 

156.197 

0 

52 

26 

5591.98 

120 

158.333 

0 

43 

33 

5468 
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5 

28 
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87.675 

2741.768 

50 

21 
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19 

19 

4526.811 
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786.798 

51 

15 

4356.238 

126 

36.712 

4910.941 

10 

19 

4348.435 
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0 
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35 

25 

4202.344 
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109.914 
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41 

15 

3249.218 
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37.361 

6307.238 

36 

10 
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14.577 
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37 

9 

2679.273 
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2 

10 
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38 

9 

2275.949 
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372.896 

27 

7 
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55 

10 
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140 
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21 

9 
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15.534 
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34 

7 
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39 

4 

1624.052 
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0 
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54 

5 
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946.245 

17 

5 

1540.868 
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23 

3 
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0 
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24 

7 
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157 
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18 

2 

776.245 
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0 
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4 

0 

0 

385 

0 

1948.62 

22 

5 

7.402 

355 

66.264 

1492.954 

53 

3 

3.22 

362 

30.262 

410 

3 

3 

152.915 

192 

49.883 

287.795 

6 

3 

3.559 

354 

26.511 

2333.972 

16 

4 

5.947 

356 

19.814 

397.322 

30 

2 

2.306 

363 

18.262 

411.345 

15 

0 

0.001 

385 

0 

417.568 

7 

0 

0 

385 

0 

416 

11 

0 

0 

385 

0 

411 

25 

0 

0 

385 

0 

415 

31 

0 

0 

385 

0 

1834.01 


218 




45 

0 

0 

385 

0 

4512.258 

48 

0 

0 

385 

0 

2887.086 

56 

0 

0 

385 

0 

2881.185 

40 

1 

1.105 

379 

0.786 

2488.567 

29 

1 

1 

379 

0.345 

1818.052 

44 

1 

1.105 

379 

0.091 

422.891 

33 

1 

1 

379 

0 

3489.579 

47 

1 

1 

379 

0 

6501.08 

13 

2 

2.836 

359 

13.306 

5845.189 

32 

2 

2 

373 

0 

426.196 

20 

5 

7.403 

355 

2.167 

430 

46 

2 

2.22 

360 

2.645 

414.865 

14 

4 

6.244 

356 

0 

429 

1 

2 

2.219 

360 

1.32 

433 

28 

4 

4.526 

356 

0.796 

416.118 

9 

2 

2.306 

359 

1.84 

4847.128 

49 

2 

2.835 

359 

1.283 

4599.95 

12 

4 

4.896 

348 

0 

5974.313 

26 

2 

16.546 

238 

3.249 

454.482 


219 



A Revision to the Revised Bloom’s Taxonomy 


Afnan N. Darwazeh 

An-Najah National University 

Robert Maribe Branch 

University of Georgia 

Key Words: Bloom, taxonomy, cognitive domain, cognition, metacognition, component display 

Introduction 

We propose a revision to the [ revised] Bloom’s Taxonomy based on the results of an extensive review of 
related literature and analyses of studies in the field of cognition and human information processing. This paper 1) 
presents the original Bloom taxonomy, 2) presents Anderson’s revision of Bloom’s taxonomy, 3) provides an 
analysis of Anderson’s revision of Bloom taxonomy, and 4) offers a revision to the [revised] Bloom’s Taxonomy. 

In early 1940’s, the cognitive researchers and psychologists had stated, based on cognitive and human 
information processing theories, that a human’s mind applies different mental processes while he or she 
manipulates, develops, insights, stores, and retrieves information (Gagne, 1977; Guilford, 1956; Lindsay& Norman, 
1977; Piaget, 1952; Rothkoph, 1966; Rumelhart, & Ortony, 1977; Wittrock, 1974). These mental processes differ in 
terms of types 1 such as memorization, comprehension, discrimination, analysis and the like, and also in terms of 
level of difficulty , such as simple, medium or complex. Accordingly, a number of instructional psychologists have 
utilized the results of cognitive research and applied its principles into field of education (Bloom, 1956; Guilford, 
1967; Gagne, & Briggs, 1979; Merrill, 1983). They aim to help teachers and curriculum developers to design 
instruction that will induce students to use different forms of mental processes ,on different levels of difficulty, so 
that the whole instructional process will be remarkably enhanced. 

One of the most prominent instructional psychologists who tried to apply the cognitive principles was 
Benjamin Bloom. Bloom (1956) published a taxonomy of educational objectives within the cognitive domain. 
Bloom and his associates classified different forms and levels of learning based on mental processes that students 
involved in while they learn. Bloom’s taxonomy gained immediate popularity; however, after almost six decades of 
using Bloom’s original taxonomy, some educators began to wonder whether the taxonomy was still valid to this age 
that characterized of lot of research and studies on intellectual skills and human thinking and learning. One of those 
educators was Lorin Anderson, a former student of Benjamin Bloom. Anderson and Krathwohl (2001) revised 
Bloom’s taxonomy to be more adaptive to our current age by proposing another taxonomy that will meet curriculum 
designers, teachers, and students needs better than the Bloom’s one. However, based on a thorough assessment we 
contend that Anderson and Krathwohl’s (2001) revised Bloom’s taxonomy still fails to match the cognitive 
psychologists recently discovered types and difficulty levels of mental processing (Bloom, 1956; Darwazeh, 1995, 
2011; Lindsay & Norman, 1977; Merrill, 1983; Tuckman, 1992;West, Farmer, and Wolff, 1991). Therefore, based 
on a current review of the literature on cognition, and particularly based on Merrill’s Component Display Theory 
(1983), further revision to the revised Bloom’s taxonomy is appropriate. 

The original Bloom’s educational taxonomy 

Bloom et. al (1956) published the taxonomy of educational objectives: a cognitive domain. Bloom et. al 
(1956) classified forms and levels of learning based on cognitive processes that learners involved in when they learn. 
Bloom considered his initial effort to be a starting point, as evidenced in memorandum from 1971 in which he stated 
“Ideally each major field should have its own taxonomy in its own language - more detailed, closer to the special 
language and thinking of its experts, reflecting its own appropriate sub-division and levels of education, with 
possible new categories, combinations of categories and omitting categories as appropriate” ( www.wikipedia.org ). 

Anderson’s Revised taxonomy as a match to Bloom’s taxonomy 

Anderson (1990), a former student of Bloom, updated and revised the taxonomy reflecting relevance to 21 st 
century work for both students and teachers as she said (Anderson& Krathwohl, 2001). Anderson changed the 
taxonomy in three broad categories: terminology, structure and emphasis (Forehands, 2005). Anderson modified 
the original terminology by changing Bloom’s categories from nouns to verbs. Anderson renamed the knowledge 
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category into remember, comprehension into understanding and synthesis into create categories. Anderson also 
changed the order of synthesis and placed it at the top of the triangle under the name of Create (Taylor & Francis, 
2002). Thus, Anderson and Krathwohl’s (2001) revised Bloom’s taxonomy became: Remember, Understand, 
Apply, Analyze, Evaluate and Create ( Figure 1). 

1956 2001 



Figure 1 . The Revised Bloom's Taxonomy by Anderson and Krathwohl (2001). 

Anderson and Krathwohl (2001) also made structural changes to the original Bloom’s taxonomy. Anderson 
considered two dimensions in the revised taxonomy instead of one, a products dimension. The two dimensions are: 
1) knowledge (or the kind of knowledge to be learned) and 2) cognitive process (or the cognitive processes to be 
used in acquiring knowledge). The intersection of the knowledge and cognitive categories form 24 separate cells as 
represented in Figure 2. Based on Anderson’s perspective, the Knowledge Dimension on the left side is composed 
of four kinds: Factual, Conceptual, Procedural, and Meta-Cognitive knowledge. The Cognitive Process Dimension 
across the top of the grid consists of six levels: Remember, Understand, Apply, Analyze, Evaluate, and Create. 
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Figure 2. Knowledge and Cognitive Dimensions of Bloom’s Taxonomy as revised by Anderson. 


Notes, Comments and Justifications on the Revised Bloom’s Taxonomy 

Based on an analysis of the Anderson and Krathwohl (2001) revision of Bloom’s original 1956 taxonomy, the 
authors recommend further revisions based six reasons: 

First. Anderson’s revised Bloom’s Taxonomy omits several points within the cognitive dimension and 
the knowledge dimension. 

Second. The organizing process is missing. 

Third. The sequence of cognitive processes should be reconsidered. 

Fourth. Remembering level should be sub-divided. 

Fifth. The difficulty level for analysis, organize, and application should be replaced. 

Sixth. A principle type of knowledge has been omitted. 
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First: The revised Bloom’s Taxonomy, which had been proposed by Anderson and her associates, omits 
several points either in the cognitive process dimension, or in the knowledge dimension. The review of the literature 
on Meta-Cognitive processes which have been conducted after the appearance of Bloom’s taxonomy in 1956 (Baker 
& Brown, 1984; Bondy, 1984; Brown, 1981; Burley, Brown, & Saunders; 1985; Darwazeh, 2004; Dermody, 1988; 
Flavell, 1976; Gagne, 1977; Gilbert, 1986; Haller, et al. 1988; Kaufman, & Randlett, 1983;Lindsay & Norman, 

1977; Paris & Winogard, 1990; Pressley, Borkowski, & Sullivan, 1985; Scarr, & Zanden, 1984; Tregaskes, 1989; 
Wham, 1987; Wade, &Reynolds, 1989) and studies on instructional design (Darwazeh, 1994; 2001, 2011; Gagne, 
Briggs, & Wager, 1992; Merrill, 1983; Reigeluth, 1983; West, et al. 1991), stated that the Meta-cognitive process is 
a process of thinking rather than a type of knowledge, with the primary definition of meta-cognition is thinking 
about thinking. 

Second: Another note on Anderson’s Revised Bloom’s Taxonomy is related to the levels of cognitive 
processes. The cognitive dimension is missing a major process that the mind performs from the earliest stages of 
human development. It is the Organizing process, which was named by Bloom as a synthesis process. Therefore, 
we cannot integrate this process under the application process and put the two of them in one category. We cannot 
also look at the organizing process as a creative process like Anderson did in her revision. The organizing or 
synthesizing process should be separated in a different category as Bloom did in his original taxonomy. 

Third: The sequence of cognitive processes should be reconsidered. Anderson and Krathwohl (2001) 
looked at the organizing or syntheses process as a creative process, which we dispute, because the synthesis process 
is easier to perform than the creation process (Figure 1). Creation needs form the person to give something new and 
original, whereas, the syntheses needs to put the elements or parts together according to a certain model or principle. 
In this case, the synthesis is a prerequisite of the creation process. Based on the perspective of synthesis as a 
prerequisite for creation, we differ from Anderson, et al. in their proposed taxonomy that considers the synthesizer 
process as an organizing process. The organizer usually comes after the analysis process. 

Fourth: The Remembering level should be sub-divided. Anderson and Krathwohl’s (2001) Revised 
Bloom’s Taxonomy did not differentiate between two levels of remembrance, (i) remember specific information and 
(ii) remember general information. Merrill (1983) has clearly differentiated in his Component Display Theory 
between the two levels based on the content type: a) remember specific information and b) remember general 
information. Merrill defined the specific information as a kind of knowledge or data which cannot be generalized 
or transferred to another situation, because, it is just valid on itself such as, dates or events, names, symbols, terms, 
labels, examples, lists, instances; whereas, the general information is a kind of knowledge that can be generalized 
and applied to more than one situation, such as concepts, principles and procedures. Merrill also claimed that 
remembering specific information is easier to recall or recognize than remember general information, and so it 
should actually come before remembering general information. 

Fifth: The difficulty level for analysis, organization, and application should be replaced. The application 
process is more difficult to perform than the analysis or organizing processes when encountering a situation for the 
first time, so the application should come afterward. The reason is, the application process defined by both Bloom 
(1956) and Merrill (1983) as the usage of previous acquired knowledge (generalities)in new situations. The 
contention here is that learners can only use or apply the acquired knowledge in a new, novel, or strange situation 
unless he or she inspects, analyzes, organizes, and re-organize its components. These processes of analyzing and 
organizing will help him or her to see where the previous learned knowledge will fit in a new situation, thus, to use 
and apply them correctly. 

Sixth: A principle type of knowledge has been omitted. The Revised Bloom’s Taxonomy is missing a 
major type of knowledge, the principle type or relationships knowledge (West, et al. 1991). Anderson (Anderson& 
Krathwohl, 2001, p. 44) put the principle under the conceptual type of knowledge, which is not true according to 
Merrill’s Component Display Theory (Merrill, 1983), West (West, et al. 1991) and Gagne (Gagne, 1977. Merrill 
(1983, pp. 287-288) in his theory had identified four types of knowledge or content: facts, concepts, principles, and 
procedures. Therefore, the principle is one type of the content. The contention here is that Bloom’s Revised 
Taxonomy (Anderson, 1990) should include the principle or relationship type of knowledge because it is different 
than the facts, concepts, and procedures. The revised Bloom’s Taxonomy should also move Meta-Cognitive 
knowledge to the cognitive dimension, because the meta-cognitive is a kind of mental process rather than a kind of 
knowledge. 
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A Suggested Revision to the Revised Bloom Taxonomy 

We propose the following taxonomy based on the originall956 Bloom’s Taxonomy and the Anderson and 
Krathwohl (2001) Revised Bloom’s Taxonomy. Our new proposed taxonomy has two dimensions: Dimension One: 
Cognitive Processes, which range from simple to complex, according to the difficulty level of the mental process 
vertically, and ranging from simple to complex horizontally according to the number of items in which a mental 
process is involved; Dimension Two according to Merrill's theory of four types of knowledge: facts, concepts, 
principles, and procedures. The content contained within the cells of Figure 3 represents the behavior that the 
learner exhibits during knowledge construction after learning has occurred. 

Accordingly, the content contained within the cells of Figure 3 represents the cognitive process (behavior) 
that the learner can exhibit during knowledge construction after learning has occurred. Check marks represent that 
students can exhibit this kind of behavior related to this certain type of knowledge, and X marks represent that 
students can't exhibit. The rationale is that once the facts are a kind of specific information knowledge which can't 
be generalized to more than one new situations, thus, the student can't comprehend, apply, evaluate, or create them. 
A learner can remember them on specific level of learning only, but not on a general level. The student also can 
discover the facts but can't create them, because they are existed in the world already. The student can also 
recognize and analyze the elements of facts either names, symbols, dates, labels, and organize and re-organize them 
based on a certain principle. For example, the student can re-order the list of names alphabetically, once recognized 
and analyzed. Finally, a student can accept the fact either he likes it or dislikes it. So he can evaluate facts, which 
can be accepted or rejected (Figure 3). 
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Figure 3. Knowledge and Cognitive Dimension of Bloom’s Taxonomy Revised by Anderson. 
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Figure 4. The revision to the [revised] Bloom’s Taxonomy 
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Conclusion 


The purpose for this proposed revision to Anderson’s (2001) revised taxonomyis to be used as in a broader 

and more comprehensive view by educators, teachers, instructors, professional trainers, and curriculum planners. 

Further, evaluators who intend to promote students’ thinking and skills in order to make them good thinkers, 

creators, discoverers, builders, and critical thinkers, will be in a better position to serve their societies and nations in 

a good and productive manner in order to live within the Information Age. 
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ElevatEd: An Innovative Web-Based Solution for 
Strategic Planning and Continuous Improvement in Schools 

loan G. Ionas, Matthew A. Easter, Blake A. Naughton 

Abstract 

Strategic planning and continuous improvement is a necessity for today’s schools. The plans cover school 
processes, school districts design, and implementation of goals towards the betterment of all school activities, with 
the final purpose of promoting and supporting learning. 

While the educational system shares much with the traditional strategic planning and continuous 
improvement process in a business environment, it also has its specific needs, governed by their primary objective to 
promote and support learning, as well as by the regulatory requirements for local and national government mandated 
reporting. In this context, ElevatEd is a novel web-based solution to Strategic Planning, Monitoring, and Analysis 
needs of schools and school districts developed at the University of Missouri by educators for educators, focused on 
meaningful use of performance and comparison data. 

ElevatEd was designed with two strategies in mind: (1) support connections between meaningful planning 
goals and objectives, first with educators’ processes and actions in their school systems and then with valid and 
reliable data that can be mined and made instinctive to teams of educators, and (2) promote a shift in the workflow 
in schools so that educators and staff begin to convene around data routinely, in order to interrogate practice and 
knowledgeably manage their own performance. These strategies are supported by six main concepts: shared virtual 
workspaces, projects, processed, data, communication, and community, which combined provide a unique 
experience and help overcome time and resource challenges. 

Introduction 

The names and works of exceptional change agents are in our collective consciousness, and the field of 
education has its own such agents (e.g. Horace Mann, Deborah Meier). While, on occasion, these visionaries change 
agents have the power to initiate and bring about change, the history of school reform in the United States public 
schools fails to provide the proof for the efficacy of such an approach to leadership in the educational landscape 
(Copland, 2003; Sabol & Pianta, 2014; Detert, Bauerly Kopel, Mauriel, & Jenni, 2000). In the vast majority of 
cases, change is brought by the concerted efforts of many individuals, by the collective actions towards jointly 
established common goals (Anderson & Kumari, 2009). 

Change is “a process, a quest for improvement rather than a search for a final resting place” (Murphy & 
Hallinger, 1993, p. 255). Today’s environment targeting performance-based accountability has institutionalized the 
idea that schools and school districts are capable of continuously improving teaching and learning as public policy. 
Nevertheless, the education research community is yet to produce substantial positive evidence of sustainable and 
replicable continuous improvement practice (Anderson & Kumari, 2009; Herman & Gribbons, 2001). 

Over the past few decades the educational system has attempted to adopt and adapt the industrial model of 
continuous quality improvement (CQI) with various degrees of success. From the industrial landscape, education 
seems to be closer to the CQI models employed in the service industry, where the CQI focuses on processes and 
outcomes as opposed to the focus on inputs displayed in CQI application in industrial production. While the 
traditional approaches to CQI in educational settings may work well for administrative tasks, its application to 
student learning has its share of issues. 

In industrial settings, one of the main goals of the CQI process is to reduce and simplify the various parts of 
the production process. It attempts to use measurements to control errors to reduce waste and costs. Traditional CQI 
focuses on well defined, predictable inputs, products or services and stakeholders at various stages during the 
production process. In education, most of the elements involved in the learning process are largely unpredictable and 
ambiguous. To adapt, educational establishments have come to view their students as customers. Furthermore, the 
traditional CQI model assumes that it is generally possible to define any given process in terms of duration, needed 
resources, and timelines. The education system does not abide by these rules, which makes it difficult to determine 
where and when in the process measurements should be taken to observe impact. Additionally, while in industrial 
settings companies have a well defined, atomic system of collecting and analyzing data that can be used to pinpoint 
specific problem areas in real time, the situation in the educational system is different. The fact that the education 
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process largely deals with unpredictable and ambiguous elements makes the analysis and interpretation of collected 
data significantly more difficult. And, while in industrial settings the many systems employed for collecting, 
analyzing and reporting data interface with each other most of the time, the systems employed by educational 
institutions do not seem to do this to the same extent. While in industrial settings organizations have control over 
their finances, processes, inputs, and outputs, given that they respect a basic set of laws and regulations, educational 
institutions have to abide by a variety of regulatory requirements from local, state, and national agencies which can 
significantly affect their funding and capabilities (Arnold & Marchese, 2011; Chappell, 1994). 

Decision making based on data can only be as good as the data collected. During any given period, 
educational institutions collect myriad of data points using the many data collection systems at their disposal. 
Nevertheless, data quality and appropriateness is another matter altogether. Questions such as “Is this the right data 
to collect?”, “Is there other data that can provide better insight?”, or “Is analyzing this data worth the effort?” are 
still much alive. Data related issues in educational settings are, largely, a consequence of the fuzziness of the 
learning process, which makes difficult to clearly define measures that work. If only for this reason, educators and 
educational institutions need to move towards selecting high-quality measures based on research. This leads to the 
need to move towards an evidence-based approach in defining the measures and data to be collected to properly 
support the continuous improvement process and linking it all with professional development efforts, evaluation, 
and reward and recognition (Benjamin, 2014; Mihalic & Elliot, 2014). 

In this context of collaborative and informed changes towards improvement, the ElevatEd model attempts 
to provide necessary scaffolds to support educators’ improvement efforts without significantly interrupting already 
established effective processes and structures. 


Baseline Case Study 

ElevatEd will first be piloted within school districts in the state of Missouri. The state was an early national 
leader in the assessment movement, establishing the Missouri School Improvement Program (MSIP) accreditation 
system and associated standards in 1990 and legislatively directing the State Board of Education to develop 
statewide performance standards and assessments starting in 1993 (Missouri Department of Elementary and 
Secondary Education, 2015). All Missouri districts are required to maintain a Comprehensive School Improvement 
Plan (CSIP), which is a five-year cycled strategic plan. Support for this improvement work comes from nine 
Regional Professional Development Centers (RPDCs). While these may seem like desirable foundations for school 
improvement, it should be noted that this is not always the case. All failing districts in the state target the same 
standards, have improvement plans, implement state assessments, and have access to professional development 
services. After 20 to 30 yeas of standards, assessment, planning, accountability, and development, no one can, with 
any certainty, point to what good any of it has done. 

Of course, Missouri schools and school districts are not short on data, though they are undoubtedly short on 
information and certainly short on knowledge (Benjamin, 2014). Missouri has over 250 school districts, each 
choosing its own system to monitor data on eleven separate state-identified accreditation subcategories, with 
massive assessment systems behind that - yet none of it is accessible by query at the district level. This then makes it 
difficult to access information and make informed decisions based on the masses of data. 

About half of the state’s districts use the student information system (SIS) developed by Tyler 
Technologies, and about half use the educator evaluation system developed by the University of Missouri (MU) 
called the Network for Educator Effectiveness (NEE) - yet these systems do not talk to each other nor is there any 
operable mechanism to getting them to talk to district plans. Further, over half of the state’s districts have fewer than 
1,500 students total, and these under-resources schools have no staff or systems to collect, process, monitor, report, 
and analyze their data. More than half of the 875,000 students in the state attend the largest 50 school districts, but 
these institutions have no way to systematically analyze, compare, or share information among districts. Due to the 
wide variance in district size and complexity, common platforms and solutions are lacking, and distinct and partially 
implemented planning/monitoring/data systems do not generate meaningful comparison data, even between like 
districts. Further, systems used by school boards or professional learning communities (PLCs) of teachers for data 
reporting, monitoring through dashboards, or short-cycle information are poorly conceptualized, underutilized, and 
have virtually no impact in the field. In sum, schools only see a rough estimate of “what” their performance are (e.g., 
aggregate test scores, financial balances, and the like), while receiving virtually no systematic understanding of 
“how” or “why” that would explain their actual performance, much less which practices got them there. 

Schools in Missouri (and most probably other states) have no system for improvement. Industrial 
management seems ill-suited for the task, and schools lack the tools to enable a system of knowledge management. 
But there is no comprehensive, web-based, secure, flexible solution for strategic planning, program evaluation, and 
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process monitoring available to school districts. Meaningful, efficient data gathering and analysis support are not 
available. There is no efficient and effective way for schools and school districts to share best practices and to 
cooperate in problem solving and planning. Schools do not need one more “silver bullet” quick fix or new program, 
but they desperately need a way to harness the knowledge and promise embedded in their exceptionally dedicated 
workforce. These factors have then shaped our approach to designing ElevatEd for use in Missouri K-12 school 
districts. 

Design Principles 

The hypothesis driving ElevatEd is that reliable change in how schools operate will only come from change 
in how teachers and administrators run our schools - specifically, with a shift towards knowledge management. 
Schools are not organized around the processes and systems that would allow educators to interrogate their own 
practices and develop customized, data-informed approaches for improving learning. Knowledge management has 
promise for changing the way we run our schools in ways that will positively affect educational achievement. 
Although there are contested definitions, knowledge management may be defined as an approach to organizational 
administration whereby tools, techniques, and technologies leverage data, information and knowledge to facilitate 
better decision making for increased organizational performance (Alavi & Leidner, 2001). 

Recognizing the vacuum, a new tool aimed at facilitating new ways to manage school improvement has 
been designed and developed at the University of Missouri. ElevatEd can be categorized a web-based knowledge 
management tool for school improvement and is a concept with ambitious scope - its purpose is not merely to enable 
the use of data to inform planning and decision-making, but also to catalyze a wholesale change in the daily 
workflow of educators. 

ElevatEd was designed with two strategies in mind: (1) support connections between meaningful planning 
goals and objectives, first with educators’ processes and actions in their school systems and then with valid and 
reliable data that can be mined and made instructive to teams of educators, and (2) promote a shift in the workflow 
in schools so that educators and staff begin to convene around data routinely, in order to interrogate practice and 
knowledgeably manage their own performance. 

Data is at the center of the the ElevatEd tools and was designed around a three-tiered system (Figure 1): 

1 . Common data, mandated by the state for all districts to collect; 

2. Comparable data, such as assessments, surveys, and other instalments used by many districts and/or 

supported by research; 

3. Customized and developed uniquely by individual districts. 

The overall design and data are emphatically 
not meant to define performance measures for 
districts, nor to push them toward one-size-fits-all 
approaches to planning and performance 
measurement. The marketplace already includes a 
sizable number of such prescriptive approaches. 
Nevertheless, it is our view that these externally 
designed planning and data system tools do not honor 
a key tenet of knowledge management that privileges 
the professional expertise of educators themselves, and 
therefore requires them to develop their own 
connections among their processes, performance, and 
plans. 

Informed by the Baldridge Performance 
Excellence Program (NIST, 2010), national and state 
specific guidance (Missouri Department of Elementary 
and Secondary Education, 2015), school developed 
approaches, and combined expertise and experience 
and researchers and practitioners, ElevatEd uses the 
Figure 1. Conceptual overview of the ElevatEd ecosystem. metaphor of Virtual Meeting Rooms to support every 

aspect of growth through planning, including, but not 
limited to, needs analysis, development, 

implementation, monitoring, and reporting. The objective is to provide a collaborative environment for schools and 
school districts that support both plan development, implementation, and monitoring and a community that shares 
best practices. 
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Through its design, the ElevatEd application 
seeks to alter the workflow behavior in educators and 
seeks to become the default, social way to gather with 
colleagues around data. One of the fundamental design 
tenets was to reduce the potentially overwhelming 
cognitive load of all the data, information, and processes 
that constitute a robust system of educational 
improvement. A simplified design that conceptualizes 
educators’ work as taking place in one of four types of 
rooms (virtual workspaces): board room, conference 
room, team room and individual room was implemented 
(Figure 2). 

Districts organize themselves in a myriad of 
ways, and this flexible structure allows ElevatEd users to 
initiate “rooms” that align with their organizational 
structures and strategic purposes. The objective is then to 
not provide the kind of professional development that 
preaches an intervention and then expecting educators to 
rally around yet one more program or initiative. Instead, 
the deployment strategy is predicated on a ground up connection to the already existing work requirements and 
workflows, and to provide illustrations and assistance with enhancing those practices. ElevatEd is designed to move 
existing work out of a compliance/accountability frame into a responsibility frame. 
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Individual Room 
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Actions j ’ 


Actions "y 





Figure 2. Conceptual view of ElevatEd workspace 
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Key Features 


ElevatEd is built upon six main pillars: shared virtual workspaces, projects, processes, data, 
communication, and community, which combined provide a unique experience and help overcome time and resource 
challenges. 
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Figure 3. Shared Virtual Workspaces (Rooms) 


Shared Virtual Workspaces ( Figure 3). Using the concept of Virtual Rooms, ElevatEd provides the means 
to organize the virtual workspace. Board, Conference, Team, and Individual Rooms, each providing room-specific 
functionality, can be created and organized in any desired configuration, to match school and school districts needs 
and workflows. The Virtual Room is one of the two main gravitational poles of the application. It was designed to 
provide a common workspace for people with shared interests and is one of the three layers defining access 
permissions. Besides the room leader, which is the one who initiated its creation, application users become room 
members by invitation, which can be accepted or declined. The shared workspace provides access to functionality 
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related to monitoring live projects, data input, creation of new projects, reporting, communication, and workflow 
definition through the assignment of component coordinators and implementers for live projects. It is also the point 
of access to the project dashboards. 
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Figure 4. Project workspace. 


Projects (Figure 4) are the virtual representation of plans. The traditional plan components, Plan, Goal, 
Objective, Strategy, and Action are all available. For maximum flexibility, each project can represent either a fully 
developed plan or only a subsection (e.g., only working on a single Goal). The project is the second gravitational 
pole of the application. It is the glue that holds together data definition, implementation schedules, checkpoints, and 
to-do lists, which are defined for each component individually. To provide flexibility, the only hard dates in a 
project’s schedule are those defining the timeframe’s start and end dates. From that point on, data collection is 
defined through frequency of input, as a multiple of a predefined base period (e.g., x number of days). To determine 
progress, a set of checkpoints can be scheduled for each component, also defined as multiples of the same 
predefined base period of the input. 
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Figure 5. Representing processes through work assignment. 
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Processes (Figure 5). Represent the school and school district’s internal delegation and assignment 
workflows. In combination, they define a chain of ownership and control over the various component of a project. 
Delegation allows a user to transfer ownership and control of a project component to another user, in a different 
workspace (room). Assignments are designed to divide the workload between members of the same workspace 
(room). Two delegation/assignment contexts are supported: one for creating a project, supporting a one-to-one 
relationship, and a second for enacting a project, supporting a one-to-many relationship. 

Data (Figure 6). Two categories of data are used: (1) data generated for specific planning activities, 
internal to and collected by the ElevatEd application and (2) data generated by other school information systems, 
collected through a third party application (Ed-Fi) (Ed-Fi Alliance, 2015) and made available in summary form, 
through Key Performance Indicators (KPIs), inside the ElevatEd application. 

Internal ElevatEd data collection is supported by a highly flexible data definition and collection system, 
which provides the foundation for monitoring and reporting. Multiple types of data collection items are made 
available. Enhanced flexibility is provided by the ability to define a data collection item at a component level and 
require it to be collected at a subcomponent level. For example, one could define a data collection item for a goal 
and require it to be collected for all actions related of that goal. In addition, data collection items can be defined for 
each type of plan component and required on an organization-wide bases, to be collected alongside project specific 
data. Furthermore, sets of data collection items can be defined at school or school district level, using the 
administrative interface, and made available to be copied in individual projects. 
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Figure 6. Definition of data collection items in the Project workspace. 


To support the continuous monitoring and improvement process, the ElevatEd application will also provide 
access to indicators and metrics generated from data collected from third party applications such as school 
information systems or state education agencies. The Ed-Fi Alliance data warehouse and software applications will 
be used for this purpose (Ed-Fi Alliance, 2015). 

Communication (Figure 7). Covers multiple context-dependent communication options, ranging from room 
announcements from room leaders, to group and/or personal messaging, e-mail, file sharing, discussion boards, 
notifications, and chat, all aimed at supporting virtual collaborative work. All communications are stored and made 
available to the users through search or for browsing. 
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Figure 7. Room discussion board and other communication options. 

Community’ (Figure 8). Through sharing, both within and between schools and school districts, ElevatEd 
was designed to provide a platform for knowledge transfer. A dedicated exploration interface was designed to 
provide users with the ability to explore their own organization projects and to share their work with others. Best 
practices, small or large projects, or entire, multi-year projects, can be shared. Furthermore, research/evidence-based 
approaches to planning can be designed by experts or teams of experts and made available to all members of the 
ElevatEd community. Community support is not limited to the sharing of projects. Sets of data collection items, lists 
of tasks and to-dos can be defined by the application managers based on research or best practices and made 
available to the entire community. Similarly, each school or school district has the ability to define its own lists and 
make them available to all their members. 
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Figure 8. Explore projects view. 


Summary and Future Directions 

With its first version ElevatEd attempts to provide an answer to the existing educational planning and 
improvement issues revealed by research studies and practice. Focused on using data as the foundation for evidence- 
based planning for improvement, ElevatEd was designed to promote the shift towards knowledge management while 
interfering as little as possible with the already established school and school district processes and structures. It was 
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designed as a flexible tool that can be easily adapted to existing patterns while encouraging best practices through 
sharing. 

While currently in beta version, the feedback received has been encouraging. Extensive usability and user 
testing is ongoing. The results will be used to improve the user experience and enhance functionality. Further 
improvements based on user feedback and research as well as the evolving understanding of what schools and 
school districts need to implement effective improvement processes are already in the planning stage. 
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Defining Backchannel 

Although there are several definitions of backchannel, these definitions vary widely based on the context of 
use. For example, a linguist would label utterances such as “uh,” “um,” and “uh-huh” as backchannel 
communications; whereas, a lawyer would potentially see a backchannel as an unofficial means of communication 
between concerned parties (Fredrick, 2013; Ross, Terras, Warwick, & Welsh, 2010). For the purposes of this study, 
a backchannel is an informal secondary or background communication channel that exists while a formal front 
channel speaker, lecturer, or panel is presenting (Cogdill, Fanderclai, Kilborn, & Williams, 2001; Du, Rosson, & 
Carroll, 2012; Farnham, Brown, & Schwartz, 2009; Kellog, Erickson, Wolf, Levy, Christensen, Sussman, & 

Bennett, 2006; McNely, 2009; Ross et ah, 2010; Yardi, 2006). 

Backchannel Platform: Edmodo 

An example of a backchannel platform is Edmodo, which was the communication channel used for this 
study. This platform was chosen for several reasons. First, Edmodo allows conversations to be threaded, which 
allows greater discussion about a specific subject within the backchannel (Cerratto, 2001). Secondly, Edmodo 
demonstrates learnability, since it closely resembles commonly used social media (i.e., FaceBook), and efficiency, 
since comments are posted quickly and easily (Gehlen-Baum, Pohl, & Bry, 2011). Thirdly, Edmodo allows the 
audience to be ‘wired’ to each other continuously throughout the professional development. Additionally, all 
conversations are recorded and can be reviewed by all participants. Both of these characters are important for an 
effective backchannel (Jacobs & McFarlane, 2005). Finally, since the ‘instructor’ of the Edmodo room sees all 
conversations, even those marked private, there is a reduction in the off-topic friend-chatter, which could potentially 
reduce the primary on-topic communication (Nobarany & Haraty, 2009). 

Benefits of Backchannels 

A recognized problem with traditional teaching and professional developments is the large amount of 
passivity by the learners (Ebner, 2009; Du, Rosson, & Carroll, 2010; Gehlen-Baum, Pohl, Weinberger, & Bry, 2012; 
Pohl, Gehlen-Baum, & Bry, 2011). Additionally, learners may feel uncomfortable asking questions because they do 
not want to interrupt the lecturer, they do not want to be perceived as incompetent by their peers, they may have a 
shy nature, or they may not know enough about the subject to construct meaningful questions (Bry, Gehlen-Baum, 

& Pohl, 2011; Du et al., 2012; Ebner, 2009; Fredrick, 2013; Gehlen-Baum et al., 2012; Nobarany & Haraty, 2009; 
Pohl et al., 2011; Yardi, 2006). By offering communication through a backchannel, there is an opportunity to 
participate in the learning process through interacting with peers without the disruption of the presentation, as well 
as reducing apprehension about asking questions and giving feedback (Boyle & Nicol, 2003; Cogdill et al., 2011; 
Gehlen-Baum et al., 2011; Pohl et al., 2011; Yardi, 2006). 

Importance of Group Communication 

Researchers have suggested that group communication in the backchannel has the potential to increase 
learning (Boyle & Nicol, 2003; Du et al., 2010; Kearns & Frey, 2010; Yardi, 2006). Yardi (2006) suggests that 
learners become involved in a community of practice with potential for situated learning when using a backchannel. 
Additionally she points out that there is an opportunity for peer-to-peer learning through the communications, 
interactions, and support provided by the peers using the backchannel. This interaction also provides the opportunity 
for active and collaborative learning (Pohl et al., 2011). 
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Furthermore, Kearns and Frey (2010) suggest that participation in a community of learners provides both 
academic and social benefits since participants provide social support as well as providing multiple perspectives and 
learning strategies. Other research suggests that the construction of knowledge occurs when learners actively engage 
the concepts they are learning with discussion, debate, questioning and explaining rather than passively receiving 
knowledge from a speaker (Boyle & Nicol, 2003; Du et al., 2010; Kellogg et al., 2006). 

Why Professional Development 

Cerratto (2001) points out that many teachers have busy schedules and often need time and opportunities to 
talk and connect with colleagues. By participating in a backchannel, teachers can learn about colleagues, find like- 
minded peers, build trust and a sense of community, explore pedagogical practices, reflect on the technology used, 
share resources and ideas, and discuss school culture (Bates & Wilson, 2012; Cerratto, 2001; Farnham et al., 2009; 
Kellogg et al., 2006; McNely, 2009). In short, they can communicate, collaborate, and learn all in one. 

Unfortunately, sometimes communication may prove challenging, given the audience. For example, 
because the room is full of peers, some participates may edit and curtail their comments (Cerratto, 2001). Moreover, 
participants may self-censor their answers due to the presence of certain peers, like administrators (Cogdill et al., 
2001 ). 


Methods & Results of Backchannel Use in a Professional Development 

The construction of this study was designed with four primary questions in mind: 1) Do participants use the 
backchannel at all? 2) Do participants use the backchannel for learning and/or building a community of practice? 3) 
Are their specific demographic groups who are more prone to use the backchannel for sharing and learning? 4) Are 
the participants “ok” with the concept of backchanneling? For this last question, the participants were asked 
questions similar to the “Diffusion of Innovations” characteristics of an innovation likely to be adopted (Rogers, 
2003). Consequently, questions asked for opinions about the backchannel and its relative advantage, compatibility, 
complexity, trialability, and observability. 

Four professional developments, over two years, were given as optional pre-conference workshops to 
educators of all levels (i.e., P-12, higher education, adult basic education) who wanted to learn how to incorporate 
more technology in their lessons. The first pair of workshops took place in San Antonio, TX, in 2013. The morning 
session had 22 participants and the afternoon session had 17 participants. The second pair of workshops was in 
Orlando, FL, in 2014. The morning session had 32 participants and the afternoon session had 24 participants. 
Participants were required to bring their own digital devices and to have registered with Edmodo before attending 
the workshop. At the beginning of each professional development, the concept of a backchannel was explained and 
Edmodo was demonstrated. Participants were then asked to post a response to a question posed by the facilitator of 
the workshop to familiarize participants with posting. Demographic data (i.e., gender, age, ethnicity, teaching 
experience, highest degree completed, grade level taught) about each participant was collected and correlated to the 
number and types of comments that each participant made on the backchannel in every workshop. Comments were 
coded as ‘helpful’, sharing or contributing knowledge to the community, or as ‘chatter’, friendly banter, or 
conversation outside the scope of a professional or educational setting. 

With regards to the first question about backchannel use, out of the 95 total participants, 61 individuals 
(64%) posted in the backchannel at least once (see Table 1 for all data). In regards to the second question about 
using the backchannel as a community of practice, 49 (52%) individuals used the backchannel to share or contribute 
their knowledge to the community of participants. In reference to the third question about who, demographically 
speaking, uses the backchannel, there were some trends in the 49 individuals who used the backchannel as a 
community of practice. For instance, of those 49 individuals, all of them (100%) were over the age of 40 and had 
more than 10 years of experience in the field of education. Most of this group (67% of the 49 individuals; 33 people) 
had Masters degrees. Additionally, most of this group (67% of the 49 individuals; 33 women) was female. Also, just 
over half (57% of the 49 individuals; 28 people) of this group was district-level employees or people who were no 
longer in the classroom. 

For the fourth question, if participants are “ok” with the concepts of backchannel, all 49 individuals (100%) 
who were using the backchannel as a community of practice thought the relative advantage, compatibility, 
complexity, and trialability worked for adopting backchannel use in education. When it came to observability, 26 
individuals (53% of the 49) had seen their peers using backchanneling in an educational setting prior to this 
workshop. These results suggest that backchanneling may coincide with the diffusion of an innovation as suggested 
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by Rogers’ theory (Rogers, 2003), but no solid conclusions can be drawn out of this very small sample. Perhaps with 
more data, definitive conclusions could be extracted. 

Table 1 . Overview of data collection from backchannel workshops 


Session 

Number of 
Participants 

Total Number 
of Comments 

Number of 
participants that 
used 

backchannel 

Comments coded 
as Community of 
Practice (COP) 

Number of 
participants who 
commented as COP 

AM 2013 

17 

38 

12 (71%) 

27 (71%) 

12 (71%) 

PM 2013 

22 

19 

16 (73%) 

16 (84%) 

16 (73%) 

AM 2014 

32 

58 

17 (53%) 

29 (50%) 

17 (53%) 

PM 2014 

24 

51 

16 (67%) 

27 (53%) 

16 (67%) 

Totals: 

95 

166 

61 (64%) 

99 (60%) 

49 (52%) 


Tips for Using a Backchannel Based on These Experiences 

Those people who would like to use a backchannel should thoroughly explain the purpose of the 
backchannel, perhaps even describing some of the benefits including a couple of scenarios, prior to use. If 
participants are on the same page as the presenter, then backchanneling is less likely to be perceived as some 
extraneous “busy” work. Moreover, if everyone understands the backchannel’ s purpose, there could be more 
participants posting and those posts could be more for learning and sharing, or, in other words, the quantity, as well 
as the quality of posts inside the backchannel, would increase. Another way to increase the quantity of posts is to 
have all participants post something in the backchannel before the front channel begins. For example, you can ask a 
question (i.e.. What is your favorite educational app?) and have the answers posted in the backchannel. Participants 
would leam how to post and also have an incentive to read the backchannel. Another way to increase the quality of 
comments in the backchannel is by setting up a protocol or etiquette for comments before using the backchannel. 
This effort can create an atmosphere for constructive feedback rather than bullying or snarking. 

If possible have a ‘Google Jockey’ or someone who monitors the backchannel. This person can notify the 
speaker of questions, can look up websites and/or Internet links, and can Google answers to some specific questions. 
This person can also monitor the backchannel in case bullying or snarking becomes a problem. 

If backchanneling doesn’t work perfectly or as well as expected, don’t give up. Perhaps try different 
platforms like Twitter/Twiducate, Chatzy, Chatterous, Cel.ly, NeatChat, Stinto, or Google (i.e., Groups, Hangout, 
Wave). Also, go back through the backchannel after the presentation to see if you can spot the problems. Learning 
from the backchannel may help you identify problems in the front channel (i.e., presentation, activities, 
demonstrations) as well. 

In watching the backchannel experiences of others, there is two more pieces of advice to offer. First, don’t 
rely on the backchannel to engage the audience. The presentation and other front channel events should be the focus 
of the presentation or the participants might not pay attention to the front channel (and maybe not even the 
backchannel). Second, be wary of posting live backchannel feeds on big screens in the room. This can become a 
distraction from the speaker and may even eclipse the entire front channel. 
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Abstract 

Failure is an expected part of games and simulations; however, this loss is also an expected part of the 
learning experience rather than an end result. In contrast, many traditional school settings use failure, but only as an 
end result. Students of a graduate educational games and simulations course were polled before and after completing 
the course to determine if there was a change in their opinions about the place of failure in education. Before 
beginning the course, 82% of the 22 students in the course thought of failure as a consequence of not learning the 
lesson or, in other words, as an end result of the learning process. At the end of the course, when the 22 students 
were polled again, 91% of the 22 students shifted their opinion by thinking of failure more as a part of the learning 
process rather than just an end result. 


The Importance of Failure 

Failure is an important part of our economy, our government, and the learning process (Albert, 2013). For 
example, failure weeds out inferior products or inefficient suppliers. Also, policies and/or leaders can be deemed as 
failing by voters who can, consequently, change their support to more promising platforms and candidates. 
Additionally, most people would agree that experiencing failure has taught them valuable life lessons. In short, 
failure often makes progress possible; therefore, failure is not necessarily an end result, but part of the process of 
living and learning. 

When a person considers failure as just a shortfall of reaching the objective, it could be said that he or she 
is really looking at failure as an opportunity for learning. The gap between reaching the objective can be analyzed 
and evaluated so that strategies can be adjusted for the next attempt. For instance, Thomas Edison, who, reportedly, 
had 10,000 failures before inventing the light bulb, embraced each failure as a step in the learning process that 
brought him closer to success (Bartes, 2009). 


Controlled Failure 

Many innovative companies encourage and fund experimentation rather than punishing or failing to reward 
employees who risk potential failure (Cannon & Edmondson, 2005). Additionally, some organizations generate 
controlled failures to simply analyze what works and what does not so that new products, services, and innovations 
may be identified. A huge advantage of this type of experimentation is that these failures can take place in 
simulations or off-line so that there is relatively little cost. 

Failure in Games 

Games offer a similar environment where it is not only safe to fail, but that a player learns from every fail. 
Becker (2008) goes a step further stating that gamers become persistent in playing since it is commonly known that 
there is a way to win. This knowledge encourages another try after every fail. This experimentation and fail-safe 
environment of games may account for part of the motivating factor of games. McGonical (2011) insinuates that 
there is no real comparison between games and reality. She suggests that games eliminate our fear of failure, 
improving our chances for success; whereas, reality is just hopeless. McGonical (2011) goes on to point out that 
although no one likes to fail, gamers can spend 80% of their play failing (i.e., run out of time, lose the fight, don’t 
solve the puzzle, die). She hypothesizes that positive failure feedback in a well-designed game is the reason. If a fail 
seems random or passive, the player could lose optimism and/or a sense of purpose. But, she argues that feedback 
that can be incorporated into a learning experience is valued and applied on the very next try. 


238 



Gee (2006, 2008) has a similar hypothesis in that good games use failure as a learning experience. But, he 
adds, the price of failure is also lowered in good games. For example, very often, good games will not make a player 
start completely over when he or she fails. Gee (2006, 2008) suggests that all of these features of failure in games 
allow players to take risks that might be too costly elsewhere, like classrooms, where failure is often seen as an end 
result. 


Teaching Failure 

During a graduate level course on the incorporation of games and simulations in education taught at 
Morehead State University (MSU) during the Fall semester of 2013, students were asked their opinion of failure 
before and after the coursework. Students were all educators, school teachers to trainers, of varying stages, from 
preservice to inservice, and levels, preschool to adult. Before the coursework began, most of these students 
considered failure as an end result. Out of the 22 students enrolled: 12 (55%) thought of failure as a consequence or 
punishment for not doing better; 6 (27%) thought that it would be a small part of the learning process because 
students would learn the lesson from the end result (i.e., from the punishment of bad grades); and only 4 (18%) 
considered it a part of the learning process. Overall, 82% of these students thought failure was an end result, more of 
a punishment or consequence for not doing better, in the learning process. 

During the MSU graduate course, students were exposed to the concepts, characteristics, and qualities of 
good games and simulations. The fail-safe nature of games and simulations were part of the lessons (Gee, 2006, 
2008; Gredler, 2004; Juul, 2009; McGonical, 2011; Simpson & Stansberry, 2008; Squire, 2005). Assignments in the 
16 week course included: educational game evaluation (choosing two educational and relevant games to evaluate 
and deciding how to evaluate them), designing a virtual world field trip for their learners, a comparison between 
instructional design and game design, a proposal for an educational game design (with storyboards) relevant to their 
practice as educators, and creation of a game evaluation rubric to build a repository of educational games relevant to 
their practice as educators and designed to share with their peers. 

When the 22 students were polled at the end of the course: only 2 (9%) thought of failure as a consequence 
or punishment for not doing better; only 2 (9%) thought that it would be a small part of the learning process because 
students would learn the lesson from the end result (from the punishment of bad grades); and the other 18 (82%) 
considered it a part of the learning process. Please note that the 2 students who polled as ‘failure as an end result’ 
had originally marked this category in the pre-course poll, so their opinions had not shifted. The 2 students who 
polled that it was ‘a small part of the learning process’ had shifted from the ‘failure as an end result’ category, so 
these two did change their opinion when the class was over. Therefore, this translates into 2 (9%) people who 
stayed with same opinion while 20 (91%) students shifted their opinion on the importance of failure as a part (not 
just an end result) of the learning process. See Table 1 for some of the students’ comments. 


Table 1 , Selected comments from the students 

Timing 

Student (before or after the Comment 

course content) 

A Before “During lessons, yes. During an exam, no. If we create a safety net for 

children, where failing while learning is ok, there is a potential for them to take 
more risks without reprisal. That being said however, I am a harsh critic of 
“falling forward,” where children who should not pass, are moved forward 
without earning the progression.” 


After “Failsafe is good, or can be good, in the right set of circumstances. Flowever, 

at the end of the day we have to draw the line in the sand somewhere. We 
have to let them fail at some point. If we don't let them fail in the short term, 
then we fail in the long term.” 


B Before “I do not believe students should be allowed to fail. In fact, if a student does 

poorly, I first try to evaluate where I let the student down. If I am monitoring 
every student every day, then no child should succumb to failure.” 


After “How we view and respond to our failures is key to recovering and moving 

forward. I believe no matter the case, as an educator and an advocate for my 
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D 


students’ success, failure should not be a road block. I will strive to provide a 
failsafe environment that is rewarding and encouraging to all students.” 

Before “I do not think students should be allowed to fail without consequence. I think 

that students should be held accountable for their own learning. When 
students refuse to work, I believe there should be a consequence. On the other 
side, I believe teachers should be given time to help students that need extra 
time and support in order to be successful.” 

After “I originally stated that I didn't think students should be able to fail without 

consequence. However, now 1 think that it is VERY important for students to 
be able to try new ideas and see if they are successful or not. In a gaming 
environment, being able to start over and with the new knowledge gained is a 
very powerful motivator and will help students to try something after failing.” 

Before “This is a hard one to answer. The teacher in me does not want her students to 

fail. When they do it tells me that I need to re-teach and look at how I 
presented the material to the class, or that some need more one on one time to 
understand the material. Then I feel like we need some failure in our lives so 
we can grow and learn that it is okay to make a mistake. When we fail we are 
challenged to try harder and to improve our best. If we never fail we never feel 
like we need to be better.” 

After “I think that it helps with perseverance and self-confidence - not to see 

themselves as a failure if they do not succeed on the first attempt. Now, 1 have 
changed my homework policy so students can make multiple attempts to 
achieve success. Students are doing better on homework and I am getting 
more CORRECT feedback from their work, than just a grade. Students are no 
longer copying each other, because they know that it doesn't count for a grade 
and want to see what they know!” 


Conclusion 

To some, failure can be considered an important part of living and learning. For example, many companies, 
practice controlled failures or establish fail-safe environments to generate innovation. Without trial and error 
procedures, where would science and technology be today? And yet education has slowly shifted from an 
environment where it is safe to tiy and fail while learning to an environment where fail is a dreadful word that is 
spoken in a hushed tone. With the high stakes testing students and educators are exposed to on a daily basis, it is no 
wonder that to them failure has become synonymous with punishment or consequence - in other words, a negative 
result. 

On the other hand, anyone who has ever played a game will tell you that he or she will start the game 
expecting to ‘die’ and that the goal is to see how far he or she can make it before failing. Moreover, people want to 
play, even though they are going to die (fail) many, many times; however, with each death, they learn more about 
how to win. Since, games and simulations have been suggested as some of the best teaching tools out there, 
researchers have been trying to find the recipe for the best games, the best teaching tools, and the best ways to learn 
(Amory & Seagram, 2003; Berlotti, Kapralos, Lee, Moreno-Ger, & Berta, 2013; Gee, 2006, 2008; Gredler, 2004; 
Juul, 2009; McGonical, 2011; McManus, Ebby-Rosin, & Kurshan, 2014; Papastergiou, 2009; Pineteh, 2012; 
Simpson & Stansberry, 2008; Squire, 2005). Perhaps using gaming concepts to remind educators that failure is part 
of the learning process and not just an end result may be a step in the right direction. Although this study shows the 
potential for educators to shift their thoughts on the relationship between failing and learning, this is one class in one 
semester at one university. Further research is required to truly divine the view educators take on failure in the 
learning process. 
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Abstract 

Active learning is an instructional technique that involves students’ active participation in the learning process 
through a variety of activities. Due to the rapid development of emerging technologies, online education becomes 
one of the major approaches to learning for adult learners. Active learning strategies applied in a traditional 
classroom environment may also be applied to online learning. It is necessary to explore the application of active 
learning in accordance with the characteristics of adult learners. In addition, we need to understand how online 
learning and active learning strategies can assist adult learners in developing their competencies for sustainable 
futures and becoming successfid lifelong learners in the information explosion era. 

Introduction 

Active learning is an instructional technique that involves students’ active participation in the learning process 
through a variety of activities (Bonwell & Eison, 1991; Escribano, Aguera, & Tovar, 2013; Hativa, 2000). 
Traditional active learning methods that are usually implemented in traditional classroom environments, evolved 
away from the static lectures to a learning environment where students actively engage in the learning process 
(Strage, 2008). Due to the rapid development of emerging technologies, online education becomes one of the major 
approaches to learning for adult learners (Allen & Seaman, 2008; Parsad & Lewis, 2008). The percentage of the 
online student population has been increasing, from 9.6% in 2002 to 32% in 201 1 (Allen & Seaman, 2013). Online 
learning shifts the use of active learning for adult learners from traditional classroom learning to web-based learning 
with the use of various technologies (Muncy & Eastman, 2012; Paetzold & Melby, 2008; Shieh, Chang, & Tang, 
2010 ). 

A substantial amount of research has shown that there are no significant differences between the effectiveness 
of online learning and traditional face-to-face learning, and that the active learning strategies applied in a traditional 
classroom environment may also be applied to online learning (Allen, Bourhis, Burrell, & Mabry, 2002; Brown & 
Liedholm, 2002; Johnson, Aragon, Shaik, & Palma-Rivas, 2000). It is necessary to explore the application of active 
learning in accordance with the characteristics of adult learners. In addition, we need to understand how online 
learning and active learning strategies can assist adult learners in developing their competencies for sustainable 
futures and becoming successful lifelong learners in the information explosion era. 

Objectives/Purpose 


The objectives of this paper include: 

• We address the history of active learning based on a review of existing literature on active learning. 

• We also introduce the types of active learning strategies along with the theories that support active learning. 

• In addition, we indicate the learning outcomes that are relevant to active learning and discuss the advantages of 
utilizing active learning strategies. 
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Method and Data Source 


We reviewed articles related to active learning through EBSCO. Key words were utilized including active 
learning, active strategy, adult learning, and online learning. Articles that are not relevant to the topic were removed. 

Findings 


An Introduction to Active Learning 

Active learning is a student-centered approach to learning that focuses on students' active involvement in the 
learning process (Deneve & Heppner, 1997; Escribano, Aguera, & Tovar, 2013; Mumoz, Martinez, Cardenas, & 
Cepeda, 2013). Bonwell and Eison (1991), the leaders of active learning, have contributed much to its development 
and the acceptance of active learning as a feasible approach to enhancing student learning. Student activity and 
engagement in the learning process are the two major elements of active learning (Prince, 2004). In active learning, 
students take responsibility of their own learning. Students learn best through "learning by doing," in which students 
are required to actively participate in meaningful learning activities and think about what they are doing (Gardner & 
Belland, 2012). Active learning may fit the needs of adult learners because adults are self-directed learners who 
intend to control their learning progress and prefer to work on something practical that connects life experiences and 
learning (Merriam, Caffarella, & Baumgartner, 2007). Active learning is aligned with the adult learning principles 
(see Table 1). 

Table 1. 


Alignment of the Characteristics of Active Learning with the Adult Learning Principles 


Characteristics of Active Learning 

Addressed Adult Learning Principles 

Students are more involved than in passive listening. 

Learner’s need to know; Self-concept of the learner 

Students are engaged in activities such as reading, 
discussing, and writing 

Learner’s need to know; Self-concept of the learner; 
Readiness of leam; Orientation to learn 

Student motivation is increased 

Motivation to learn 

Students can receive immediate feedback 

Learner’s need to know; Orientation to learn; Motivation 
to learn 

Students may be engaged in higher-order thinking, 
such as analysis, synthesis, and evaluation 

Learner’s need to know; Prior experience of the learner; 
Readiness to learn; Orientation to learn 


Active learning strategies encompass various activities (e.g., icebreakers, class discussions, answer pairs, one 
minute paper, cooperative learning, student debates, games, role-playing, Jigsaw, computer-aided instruction). 
Selection of appropriate active learning methods depends on the level of students and the content that is being taught. 
Learner motivation increases when active learning strategies are appropriately adopted in an instruction (Carroll & 
Leander, 2001). 

Active learning strategies have been developed as effective methods to complement traditional classroom 
learning that is primarily static lecture -based (Paetzold & Melby, 2008). Applying active learning instruction will 
help increase student interaction and knowledge retention in the traditional classroom learning environment 
(Paetzold & Melby, 2008). Passive learning, as opposed to active learning, does not require students to actively 
participate in learning activities. Instead, passive learners are instilled knowledge from the instructor through 
lecture -based instruction or receive information by reading course materials on their own (Phillips, 2005; Prince, 
2004; Wilson, Pollock, & Hamann, 2007). Fewer higher-order cognitive learning skills, such as analysis and critical 
thinking, are developed through passive learning, and lower-level learning, including rote memory and reciting, 
often becomes the outcome. Dale's (1969) cone of learning, a visual metaphor for learning modalities, indicates the 
differences between active learning and passive learning in terms of the degree or levels of learning that takes place 
with different learning modalities involved. When examining a variety of active learning strategies, we found that 
active learning strategies address the adult learning principles to a large degree (see Table 2). 
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Table 2. 


Active Learning Activities and Adult Learning Principles 


Active Learning 

Adult Learning Principles 

Activities 

Learner’s 
need to 
know 

Self-concept 
of the 
learner 

Prior 

experience 
of the 
learner 

Readiness to 
learn 

Orientation to 
learning 

Motivation to 
learn 

Ice breakers 

- 

- 

V 

- 

- 

V 

One minute paper 

V 

- 

V 

- 

- 

V 

Question/ 
answer pairs 

V 


V 

V 

V 

V 

Class discussions 

V 

V 

V 

V 

V 

V 

Student debates 

V 

V 

V 

V 

V 

V 

Role playing 

V 

V 

V 

V 

V 

V 

Games 

V 

V 

V 

V 

V 

V 

Cooperative 

learning 

V 

V 

V 

V 

V 

V 

Jigsaw 

V 

V 

V 

V 

V 

V 

Case study 
method 

V 

V 

V 

V 

V 

V 

Fieldwork 

V 

V 

V 

V 

V 

V 

Independent study 

V 

V 

V 

V 

V 

V 

Computer-aided 

instruction 

V 

V 

V 

V 

V 

V 


Note. The sign “v” refers to the activity that represents the principle, and ” indicates that the activity does not well 
represent the principle. 


Underlying Theories of Active Learning 

The claim of constructivism that people learn by constructing their own understanding and knowledge of 
the world based on previous experiences and prior knowledge serves as the foundation of active learning. In the 
view of constructivist, learning is a process of knowledge construction instead of knowledge recording or absorption 
(Anthony, 1996; Chalufour, 2014; Fiillsack, 2013). The role of learners is shifted from a passive recipient of 
knowledge to an active constructor of knowledge who builds an internal illustration of knowledge and a personal 
interpretation of experience. Learners are knowledge-dependent as they build on current knowledge to construct new 
knowledge. Moreover, active learners are often aware of the processes of cognition and can control or regulate them 
based on their needs or situations. These assumptions of constructivism are in line with the core elements of active 
learning. 

Learning Outcomes as a Result of Active Learning 

Active learning has been linked to better learning outcomes, including achievement, attitudes, and behaviors 
(Michel, Cater, & Varela, 2009; Taraban, Box, Myers, Pollard, & Bowen, 2007). Particularly, research has shown 
that active learning strategies enhance learners’ higher-order thinking, including critical thinking, problem-solving, 
synthesis, analysis, and evaluation (Bonwell & Bison, 1991; Richmond & Hagan, 2011). Development of higher- 
level thinking becomes one of the most significant aims for active learning (Pundak, Herscovitz, & Schacham, 2010). 

A wide range of evidence supports the importance of active learning in receiving higher-order thinking, and 
its superior role over traditional learning methods. The conceptual framework of active learning proposed by 
Watkins, Carnell, and Lodge (2007), which implicitly depicts the cognitive learning domain of Bloom’s taxonomy, 
provides a basis of the measurement of learning outcomes for active learning. The three distinct dimensions in this 
framework include behavioral, cognitive, and social elements (Watkins, Carnell, & Lodge, 2007). This framework 
rests on two relevant constructivist theories, cognitive constructivism and social constructivism. 
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The Advantages of Applying Active Learning Strategies 

Based on the positive learning outcomes from active learning, researchers have discussed the benefits of 
active learning (Bonwell & Eison, 1991; Phillips, 2005; Watkins et ah, 2007). In addition to its academic advantages, 
active learning has been shown to bring social and psychological benefits (Gavalcova, 2008; Slavin, 1996). These 
benefits of applying active learning include an increase of learners’ motivation to learn, self-confidence, and self- 
reliance; enhancing the opportunities to retrieve previously learned knowledge; fostering social interdependence and 
support; improving attitudes towards subject areas and student retention; and enhancing skills to collaborate, 
communicate, or interact with others (Gavalcova, 2008; Kane, 2004; Phillips, 2005; Watkins et ah, 2007). Moreover, 
active learning is found to be positively related to perceived course quality (Taylor & Ku, 2011). 

Active Learning and Adult Learners in Online Learning Environments 
Application of Active Learning among Online Adult Learners 

The active learning strategies applied in a traditional classroom may be also applied to an online course with 
adult learners, as long as the multiple learning styles of adult learners are considered (Paetzold & Melby, 2008). 

Cost, individual learning styles, instructional skill sets, and technology support are important factors that influence 
an instructor’s decision of selecting active learning strategies (Phillips, 2005). Although there are different types of 
active learning strategies, not all of them can be properly applied in online settings. Instructors need to understand 
the needs of online adult learners with different learning styles and be prepared to utilize active learning strategies to 
help each style of adult learners and enhance student interaction in online settings (Kuo, Walker, Schroder, & 
Belland, 2014; Paetzold & Melby, 2008; Phillips, 2005; Vincent & Ross, 2001). 

To incorporate active learning techniques into online learning, instructors must know the technology they 
choose for an online course (e.g., the strengths and weaknesses of technology tools, main features, potentials, 
quality), and think about whether the selected technology tool can efficiently engage adult learners in active learning 
activities, and thus, enhance student learning outcomes (Parker, Lenhart, & Moore, 2011; Phillips, 2005). Although 
the selection of proper technology tools is important for an online course with adult learners, the focus should not be 
the technology, but the development of an online course that incorporates active learning. 

Important Factors for the Design of Online Active Learning 

A proper design of active learning in online settings is necessary to facilitate student learning processes. 
Hutchings, Hadfield, Howarth, and Lewarne (2007) indicated seven principles that guide the design and 
development of active learning in web-based learning environments. Formed based on Kolb’s (1984) experiential 
learning cycle and Laurillard’s (2002) conversational framework, these guiding principles of active learning design 
emphasize the relationships between learning process and the role that teachers play to guide learners by taking both 
learner-centered and teaching-focused approaches into consideration. These principles also represent the important 
factors that instructors or instructional designers should consider when developing an online or web-based course 
with active learning. 

Challenges and Difficulties of Implementing Active Learning in Online Settings 

Although much evidence has shown the benefits of applying active learning, many faculty members are 
reluctant to utilize it in the class due to the obstacles they have encountered, especially in online courses. These 
obstacles include content coverage issues, time consumed, fears of new learning techniques, student reaction, 
teacher characteristics, technology, and pedagogical issues (Faust & Paulson, 1998; Michael, 2007). It takes too 
much time for faculty to prepare for a course incorporating active learning techniques, especially for those who 
teach a new course or who are using active learning the first time (Michael & Modell, 2003). A fear of utilizing new 
innovative learning strategies is another impediment to incorporating active learning. In addition, lack of teacher 
maturity and perceptions of colleagues may influence the adoption of active learning. In addition to the problems 
that the teachers encounter, students may also experience some barriers in an active learning class (Michael, 2007). 
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Conclusion 


We review the major trend of online learning in adult learning, the concept of active learning, and the 
application of active learning in online learning environments. Public and private institutions have been offering 
more online courses or degrees than before. Online learning provides extended opportunities for adults in the 
workforce to earn a college or an advanced degree, which helps resolve the shortage of future positions in the United 
States. Adults return to the school for many reasons, and facing the changes in life and career plans appears to rank 
the top of the list for adults to continue the education. Online learning benefits adults in several ways (e.g., 
convenience, life transitions, professional development, and increased chances for future promotions). It is important 
to design effective online learning by considering the characteristics and learning needs of adults. Active learning is 
an instructional method that can facilitate adults’ learning in online settings. It has the potential to increase adult 
learners’ levels of engagement in the learning processes and, thus, enhance the effectiveness of online learning. 

Although active learning is popular in K-12 or higher education, there is limited research of active learning in 
the adult education literature. Few researchers have addressed the role of active learning in continuing education 
from the perspectives of adult learning. The majority of active learning research was completed through case studies. 
On one hand, there is a lack of online active learning studies at program or institutional levels. It is necessary to 
include the viewpoints of the faculty, program directors, or institutional leaders for active learning. On the other 
hand, researchers should investigate non-traditional adult learners’ perspectives of attending online courses 
involving active learning strategies. In addition, this study did not draw on the literature in training. Future studies 
should extend the application of active learning to both formal (e.g., courses offered towards a degree program) and 
informal (e.g., training) learning settings, and compare the use of such instructional method in two different settings. 
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Abstract 

Competency-based education offerings are increasing in public and private colleges and universities across the 
nation. This shift in education is new to higher education, and with that, comes much ambiguity. The instructional 
design team within the University of Wisconsin-Extension’s Division of Continuing Education, Outreach, & E- 
Learning has been working for the past two years to develop a research-based, educationally sound model for 
designing and developing competency-based assessments. These efforts were a significant contribution to the first 
system-wide, competency-based initiative in the nation: the UW Flexible Option. This paper presents a synopsis of 
the instructional design team’s efforts, lessons learned, and recommendations. 

Introduction 

While competency-based education (CBE) is not new to higher education (Grant et al., 1979; Hodge, 

2007), even in medicine, where it has been growing for the last 20 years (Frank et al., 2010; Morcke, Dornan, & 
Eika, 2013), how to define it on a national level remains ambiguous for practitioners and educators alike (Hodges & 
Lingard, 2012; Pijl-Zieber et al., 2014). One of the most significant roadblocks to implementation stems from asking 
instructors to think differently about content design (Hoogveld, Paas, & Jochems, 2005). This paper presents how 
the University of Wisconsin System is approaching CBE assessments on a system-wide scale, and the lessons 
learned as collaborative work among instructors, instructional designers, and administrators continues to shift 
thinking and practice toward competency. 


Background 

In January 2014, the system launched the UW Flexible Option program, a collaboration between the UW 
System campuses and UW-Extension’s department of Continuing Education, Outreach & E-Learning (CEOEL). 
The program allows students to demonstrate mastery of a subject area and earn college credit without adhering to a 
rigid course schedule. Since Flexible Option is meant to establish valid measures of ability regardless of study time 
or particular study materials, drafting requires an approach to assessment design fundamentally different from that 
used in traditional online courses. In fact, how assessments are created and aligned with instruction and resources is 
often considered the foundation of CBE (Biggs, 1996; Tillema, Kessels, & Meijers, 2000). As CEOEL instructional 
designers have partnered with UW faculty, difficulties and opportunities that come with thinking about assessment 
in different ways have been uncovered. In collaboration with faculty, CEOEL instructional designers are designing 
a research-based process that both identifies the characteristics of quality assessments and provides 
recommendations about when to focus on each of them. 
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The Foundation for Quality Assessments: Well-Written Competencies 

A clear and concisely written set of competencies and outcomes is an essential foundation for developing 
authentic, well-planned assessments. CEOEL instructional designers, in collaboration with UW Flexible Option 
leadership, created a three-level competency hierarchy consisting of program competencies, assessment 
competencies, and outcomes (Figure 1). 


P 


Program Competencies 





c 


Assessment Competencies 



Outcomes 


oooo 


AZAZAZAZA 

Figure 1. University of Wisconsin’s Flexible Option competency hierarchy. 


Program Competencies 

Faculty begin a competency-based assessment development by defining a set of competencies that are 
applicable at a program level. Program-level competencies are intentionally high-level, intended to capture the scope 
of a degree or certificate. The language used in program competencies defines what makes a student competent at a 
program level and should identify fundamental skills that are unlikely to change over time. 


Assessment Competencies 

Each program competency is categorized further into assessment-level competencies that define what is 
being assessed. Assessment competencies are likely to appear on transcripts with grades or other designations 
indicating mastery in the same way that course names and numbers are utilized today in a traditional course 
transcript. Instructional designers also use assessment competencies to verify and test the alignment of competencies 
and assessments within the UW Flexible Option program. Assessment competencies likely experience moderate 
change over time as each domain evolves and new trends are embraced. 


Outcomes 

Outcomes are written primarily to help students identify how they will demonstrate mastery of a particular 
assessment competency. They tell students what specifically to study as they explore learning resources. Of the three 
levels, outcomes are the most likely to change over time as resources and assessment methods evolve. 

The three-tired competency hierarchy is critical to assessment design in at least two ways. First, it helps 
instructors and instructional designers define assessment scope. While program competencies are usually in place at 
the start of design, assessment authors may create the remaining two levels and assessments in tandem because of 
their push-pull nature. Each assessment is the concrete manifestation of assessment competencies and outcomes, and 
therefore sometimes helps to shape them. However, even if they are not completely fixed in advance, these levels 
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remain a boundary demarking what is and isn’t relevant assessment content. Second, the hierarchy provides a 
standard of alignment by which assessment authors can evaluate how well the skills, knowledge, and behaviors 
elicited by a given assessment method represent their intended targets. As upcoming sections of this paper discuss, 
alignment remains one of the most foundational, yet difficult, aspects of implementing competency-based 
assessments. 


Identifying Characteristics of Quality Competency-Based Assessments 

Working with a focus group of CBE experts, Baartman et al. (2006) created a 12-criteria “Wheel of 
Competency Assessment” that summarizes the qualities of high-quality assessments and depicts visually how they 
align with each other. The CEOEL instructional designers started with this list to ultimately develop a set of eight 
criteria particularly relevant to the Flexible Option program: usability, alignment, cognitive complexity, authenticity, 
fairness, consistency, fitness for self-assessment, and transparency (Table 1). 


Assessment Consideration 

Stage 

Usability 

1 

Alignment 

2 

Cognitive Complexity 

2 

Authenticity 

2 

Fairness 

2 

Consistency 

3 

Fitness for Self-Assessment 

3 

Transparency 

3 


Table 1. Eight assessment considerations for competency-based assessments. Adapted from Baartman, L. K. J., 
Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2006). The wheel of competency assessment: 
Presenting quality criteria for competency assessment programs. Studies in Educational Evaluation, 32, 153-170. 

The smaller set of criteria came out of discussions that first used the lens of germane versus extraneous 
cognitive load (Sweller, van Merrienboer, & Paas, 1998) to identify potential arbitrary barriers to student success 
(unclear instructions, complex interface design, etc.), which can be particularly important for a largely self-directed 
CBE curriculum. CEOEL instructional designers also refined the list to use language familiar to faculty that 
addressed the most common questions received during the design process. 

Drawing from existing research, CEOEL instructional designers further developed two criteria: authenticity 
and cognitive complexity. Authenticity is defined as fidelity to professional practice, but this is more 
straightforward in some domains than in others. For example, authenticity is easier to establish in a largely 
procedural domain like nursing or software coding, but more ambiguous in an area such as a freshman-level music 
appreciation course. Drawing on Gulikers, Bastiaens, and Kirschner’ s five-dimensional framework for authentic 
assessment (2004), recommendations were developed for fidelity to professional thought, hypothesizing that, for 
example, while a multiple-choice question may not be a faithful representation of practice in some domains, it has 
the potential to elicit professional thought if crafted carefully. Drawing on research suggesting that authentic 
assessments encourage self-regulation through self-assessment (Olfos, 2007), instructional designers developed 
recommendations for tools such as small formative assessments, comparison with expert answers, rubrics, and 
practice assessments. 

Cognitive complexity (Webb, 1997; Webb, 1999) is not unique to CBE, but it plays an important role when 
assessment takes precedence over instruction, as in the Flexible Option program. Drawing from Webb’s work and 
research on how even traditional assessment methods such as multiple-choice tests can elicit higher-order thinking 
(Burton et al., 1991), CEOEL instructional designers developed recommendations on how to create tasks and 
questions that require making multiple connections and synthesizing ideas. This was a familiar idea to faculty used 
to Bloom’s cognitive domain taxonomies (Bloom et al., 1956). The instructional designers simply provide strategies 
and examples that can be incorporated into instructors’ chosen assessment formats. 


251 




Assessment Development Toolkit and the Drafting Process 


After the instructional designers finalized the eight criteria and started using them, assessment authors 
became frustrated almost immediately; trying to balance all eight criteria at once can be cognitively overwhelming 
and lead to haphazard results that are strong in one criterion but not another. Therefore, instructional designers 
created an assessment toolkit that not only explains the criteria but also sequences when to focus on each through a 
three-stage assessment drafting process. 

Stage 1: Initial Planning 

The first stage, Initial Planning, involves deciding on all assessment methods for a program in order to 
establish project scope. This stage focuses primarily on the usability criteria, asking assessment authors to identify 
the assessment methods that are most useful, scalable, and maintainable over time. Assessing usability ensures that 
the benefits of the assessment outweigh the time and resources needed to create them. 

Designers encourage authors to consider several questions during the initial planning stage such as: (1) 
What method(s) will be used for assessments? (2) In formats that involve a certain number of questions or tasks, 
how many questions or tasks will each assessment have? (3) How many attempts will be allowed for each 
assessment? (4) Will rubrics be used for grading purposes and if so, will they be student-facing, faculty-facing, or 
both? and (5) Will practice assessments be utilized? 

Initial alignment of competencies and assessments remains an enduring consideration that starts during 
Stage 1 . After several months of trial and research, instructional designers at CEOEL determined it best to assess 
competencies using a one-to-one or one-to-many relationship. During Stage 1, it is important to decide whether each 
assessment competency will be assessed on a one-to-one basis, where one assessment competency is assessed using 
one assessment, or on a one-to-many relationship, where several assessments are used to assess one competency. 

Once these questions are considered, instructional designers and authors collaboratively begin developing a 
map to ensure that planned assessments align with competencies and are usable (Table 2). 


Assessment Competency 

Learning Outcomes 

Assessments 

Analyze the effect nutrition has on 
chronic disease and overall wellness 

Analyze behaviors that limit the 
progression of chronic disease 

Evaluate resultant health outcomes 
based on nutritional choices 

Case study: 

• Chronic disease analysis report 

• Analysis of current diet and 
issues 

• Create diet for the individual 
based on his or her likes and 
dislikes 

• Summary of specific health 
benefits of the new diet 

(Format: dropbox submission) 


Table 2. Example of initial assessment competency, outcomes, and assessments map. 


This initial map provides an opportunity to visually consider whether the planned methods of assessment 
are feasible considering faculty time and resources, scalability, and maintainability prior to investing time in 
assessment drafts. 

Stage 2: First Draft and Feedback 

During the second stage, authors create assessments and focus on the next four assessment criteria: 
alignment, cognitive complexity, authenticity, and fairness. Alignment evaluates how well assessments represent 
competencies, outcomes, curriculum, and content. Cognitive complexity challenges faculty to consider the inclusion 
of tasks that require higher cognitive skills. Instructional designers challenge faculty to give all assessment questions 
the “Search Engine Test.” If the answer to an assessment question can be found simply by typing the question in an 
common Internet search engine, it is likely not cognitively complex enough to evaluate a student’s true ability. 
Authenticity of assessments is important in CBE so that students can demonstrate mastery of their skills using real- 
world situations and challenges. Finally, fairness ascertains how well an assessment’s challenge lies solely in its 
target skills and knowledge, rather than factors like primary language, financial circumstances, access to technology, 
or other considerations that create a potentially unequal playing field. 
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This stage is where the bulk of the work in the assessment creation process takes place. All assessments, 
including practice assessments, and grading rubrics and schemes are created during Stage 2. Practice has shown 
instructional designers and authors the importance of not underestimating the amount of time necessary to create a 
quality assessment drafts. Instructional designers and faculty estimate the amount of time they think is needed to 
complete all assessments, rubrics, and grading schemes, and then double that amount. This is especially important if 
this is the first time either is creating competency-based assessments. Instructional designers and faculty will spend 
the bulk of their assessment creation time in stage two and should plan to collaborate often during this time. 

As in Stage 1, instructional designers work with faculty to consider: (1) How do assessments truly measure 
targeted competencies? (2) How can assessments be designed to require more than memorization and repetition, 
where appropriate? (3) How can assessments be created to mimic professional thought and practice in some way? 
and (4) How can we ensure that assessments are accessible to students of different locations, financial statuses, 
languages, and cultures? As a result of these considerations, instructional designers and faculty often revisit some of 
the decisions made in Stage 1 . 

Stage 3: Additional Drafts 

After completing first drafts, instructional designers and faculty collaborate to provide feedback and make 
revisions, thereby leading to the final stage. The focus of the additional drafting stage is on the remaining three 
assessment considerations: consistency across assessments, fitness for self-assessment, and transparency. 
Consistency across assessments ensures that established evaluation methods are consistent across assessors, 
students, and time. Fitness for self-assessment promotes student-centered, self-regulated learning. Lastly, evaluating 
the transparency of each assessment certifies that the instructions are clear and understandable to all participants, 
regardless of any external factors. 

During Stage 3, faculty and instructional designers work closely to refine the initial assessment as well as 
any assessments intended for additional attempts (e.g., assessments students will complete if they do not achieve 
mastery on the first attempt). This is the time to put the finishing touches on all assessment-related materials. 

Assessment Development Toolkit 

The toolkit developed by the CEOEL instructional design team supports faculty by outlining this three- 
stage sequence and allowing content authors to drill down into specific criteria to see general steps on how to 
proceed, supported by examples taken from Flexible Option offerings. Designers can use all or parts of the toolkit 
to structure discussions with faculty members, who can then reference it as they develop assessments. The toolkit is 
agnostic toward content and format. Therefore, it can be used in both one-on-one and group settings across content 
areas. A criterion such as “authenticity” applies equally to a multiple-choice exam, portfolio, and simulated 
interview. 

During Stages 2 and 3, instructional designers work with faculty authors to complete a 
competency/assessment alignment template more detailed than that which is shown in Table 1. The updated 
template provides a granular level of detail regarding each assessment and how it aligns with its intended 
competency. An abbreviated example of this tool is shown in Figure 2. 


Competency Set ID: HES 209 







Competency Set Title: 







Nutrition and Weight 


Assessments 





Competencies and Outcomes 

Bloom's 

Level 

# questions or tasks 
per outcome (per 

Assessment student) 

# attempts 
allowed 

Total 

questions / 
tasks needed 

Has student- 
facing rubric? 

Has grader- 

specific rubric? % of grade 

Analyze the relationship among nutrition, physiology, 
and the enhancement of health. 

* 

Multiple-Choice Exam 57 

Short Answer Exam 4 

2 75 N 

2 8 (one version N 

N 57/455 

Y 23/455 


Figure 2. Example of detailed competency and assessment alignment mapping tool. 


Some assessment authors have found the level of detail in this tool overwhelming, suggesting the tension 
between tracking the necessarily large amount of detail needed to craft and maintain a competency-based assessment 
and presenting that detail in a way that is clear and actionable. The CEOEL instructional design team is currently 
evaluating how an updated version might best serve faculty in the future. 
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Conclusion 


At the time of this writing, two iterations of the assessment drafting process have been completed across 
multiple campuses. The CEOEL instructional design team is drafting a third iteration and organizing assessment 
creation by using the toolkit as a touch point between faculty and instructional designers. The goal of our 
presentation is to openly share both what seems to be working — such as focusing on different criteria at different 
times — and where we still have questions, such as those surrounding the use of a standard template to capture all 
assessment information. 

We continue to evaluate assessments based on student feedback, student completion rates, and faculty 
feedback. Above all, we are learning the importance of challenging how both designers and instructors think about 
assessments as a prerequisite for effectively using resources like the toolkit. We share both the challenges we have 
encountered and our emerging solutions in the hope that we can provide insights for other organizations pursuing 
competency-based assessments, while also learning from those organizations. 
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Introduction 

A “hackathon” is a computer-programming event in which volunteers work intensely in small teams for a 
short amount of time to develop a program prototype. These events became widespread during the 2000s as software 
companies and other venture capitalists used them to “quickly develop new software technologies, and to locate new 
areas for innovation and funding” (Briscoe & Mulligan, 2014, p.4). Recently, their popularity is spreading 
worldwide across undergraduate institutions in computer science (Gould, 2014) as a way to give students a focused, 
“real world” experience working in a team problem-solving environment. Due to their great impact, similar 
activities have been organized in several other disciplines, including bioinformatics (Trainer & Herbsleb, 2014), 
marketing (Calco & Veeck, 2015), healthcare (Aungst, 2015), Big Data (Gould, 2014), and dance composition 
(Briscoe & Xuan, 2014). 

The most common elements that characterize hackathons are: (a) participants are organized into small 
groups that work intensely (sometimes around the clock), (b) a short time frame, usually during an long weekend, in 
which the project is taken from concept to prototype, (c) a centralized location where teams meet, work, and share 
resources and (d) support (technical, food, mentoring, etc.) provided by organizers and sponsors. 

The Ideas of March Hackathon 

The Computer Science (CS) program at CSU Monterey Bay has conducted a hackathon, called the Ideas of 
March, every spring since 2012. This event is a joint effort that also includes the College of Business and the 
Institute for Innovation and Economic Development (IIED). Local nonprofit organizations and small businesses 
from the community pitch ideas for mobile applications and then serve as clients for teams of four students who 
create a prototype app over two and a half days. Most participants are undergraduate students in the CS program, 
but the event also attracts a number of students from local community colleges; every student who applies is given a 
spot in the competition, no students are turned away Registration for the Ideas of March is free and students can sign 
up either as a team of four or individually. Students who sign up individually are placed on a team at the beginning 
of the event with an attempt to balance skill level across teams. We have chosen teams of four participants based on 
the literature about cooperative learning which shows that teams of four allow all participants to be actively involved 
in meaningful teamwork (Johnson & Johnson, 1991; Slavin, 1995). 

Since the Ideas of March is hosted at a primarily undergraduate institution, the primary focus of our event 
is learning. Prior to the kick off for each Ideas of March, a boot camp is held for students to become familiar with 
the tools needed to participate. Additional materials are provided via the event website for self-study. In addition, 
unlike regular hackathons, we have implemented a “peer-led hackathon” approach in order to enhance the learning 
experience of participants. This approach is loosely based in the Peer-led Teaching Learning (PLTL) instructional 
strategy (Gosset et al., 2001) in which students with a higher level of knowledge and skills about the content covered 
assist groups of students, acting as tutors or mentors. This is accomplished by having advanced students and 
practicing professionals serve as mentors to the student teams competing. 

Additional instructional strategies that we have implemented in our hackathon include, among others: 
Project-based learning - students work together for three days to solve an authentic problem; Legitimate peripheral 
participation (Lave & Wenger 1991), - Freshmen are able to interact with students in higher grades and form part of 
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a community of programmers, community college students are encouraged to participate as part of teams that 
include upper division undergraduate students; Social Learning Theory (Bandura, 1997) - Mentors serve as role 
models, we aim at recruiting diverse mentors (female, male, Hispanic, etc.); Cooperative Learning (Johnson & 
Johnson, 1999) - Teams share a common goal and they need each other to succeed; Gamifi cation - teams compete 
against each other and need to complete a task within a specific time frame in order to win the contest. Aligned with 
Vygotsky’s Zone of Proximal Development, mentors help in overcoming critical programming obstacles that teams 
might face and that they would be unable to solve on their own or need a significant amount of time and effort. 

Event Logistics 

On average, approximately 30 students participate in each Ideas of March hackathon. The event lasts three 
days in which meals are provided for free to all participants and all mentors and students receive a t-shirt or other 
memento of the event. In addition to the hackathon itself, several workshops (called bootcamps) are held prior to 
the event to acquaint students with software and programming tools needed. All of these events are possible due to 
sponsorship, both internal university funds and support from external funders such as Google and AT&T. The Ideas 
of March is held each year in March and planning typically begins the preceding November. Student registration, 
mentor participation, and app idea submission from local organizations is solicited using online forms. 

The first day of the event, the representatives of the local businesses and nonprofit organizations interested 
in having an app developed pitch their project idea to all participants. Every year, there have been more proposed 
projects than teams, therefore some projects are not developed. Each team of participants selects five projects they 
would like to work on, in order of preference. The organizing committee assigns projects to the teams based on 
interest. Each team then meets the community project client for about an hour, availability during the weekend of 
the hackathon is a requirement to pitch an app to the students. The client contact information and availability during 
that weekend is also provided to each team, in case there are any follow up questions or concerns. Teams start 
designing and prototyping the project during the remaining hours of the first day. Each team is assigned a classroom 
with whiteboards and markers that they can use for brainstorming ideas. Also, all classrooms have electrical outlets 
and wireless Internet access for participants’ laptops. Teams without personal computers are given access to campus 
computer labs for the duration of the event. 

The second day of the event, a Saturday, is mainly used for programming and testing. Most participants are 
already familiar with online version control tools, which allows them to code collaboratively without affecting each 
other’s progress. For students who are less familiar with version control, a free workshop on the open source version 
control tool Github is provided prior to the event. The second day is the most important for mentor participation. 
Mentors help solving programming issues and providing suggestions about alternative ways to approach a problem. 
Mentors are available for groups on an as-needed basis and faculty members proactively check in with the groups to 
see how they are doing and to help head off any potential issues. Due to University regulations, faculty or staff 
members are present for all hours of the hackathon. 

In the last day of the event, teams present their prototypes in a reception that is open to the public. A panel 
of three to five judges evaluates the prototype apps that were developed in terms of scope, design, and performance. 
The presentation is an essential part of this process each student team must present the goals of the organization they 
are partnered with, clearly explain the problem/need for a mobile application, demonstrate their solution, and 
explain how their solution fills the need of the organization. Each team has just about seven minutes to explain the 
problem they solved and to actually show the app they created. The teams develop and rehearse the presentation for 
at least an hour and mentors help with preparations and provide feedback on a required presentation rehearsal. After 
all the groups have presented, student awards are presented and a reception is held. 

A Community-based Event 

Our hackathon follows a project-based learning model: learning is organized around a challenging, 
complex, and authentic problem that requires collaboration, reflection, and group skills (Thomas, 2000). The 
projects are proposed or suggested by local non-profit organizations, educational institutions, and small business 
who need a mobile application that will help them solve a specific problem. Faculty organizers work closely with 
community partners to ensure that expectations are clear on all sides. Representatives from participating 
organizations must agree to be available for student questions for the duration of the event so that students can 
receive feedback on features as they are developed. Likewise, partners must agree to have the code generated 
released under a creative commons license and are advised that the apps developed are early prototypes and might 
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not be fully functional. While the hope is that the products of the Ideas of March will be useful for local 
organizations, the primary focus of the event is on education. 

Participating organizations are informed that, since the Ideas of March is just a weekend event, it is not 
possible for students to complete a fully functional mobile app in such a short time. The outcome of the weekend 
hackathon is typically a proof-of-concept prototype, and students work with community partners to determine which 
features to implement. One criticism of hackathons is that the projects are frequently abandoned. We try to avoid 
this as much as possible by giving community partners and students different options to continue their collaboration 
beyond the event. While there is no formal commitment that the collaboration will exist beyond the end of the Ideas 
of March, several avenues are provided for students to continue to work with community partners with varying 
levels of faculty facilitation. We have had some instances where students from Ideas of March teams get hired by 
their community partner to complete their app as an internship or part-time employee. In other situations, students 
are able to continue to work on a project as their senior capstone or as an independent study for college credit. 

Students’ Perspectives of their Participation in the Hackathon 

A day after the 2015 hackathon finished, a link to an anonymous survey was sent to all 52 students who 
participated in the event and 23 of them (44%) submitted it. From their responses, we observed that: 

57% of the students who participated in the 2015 Ideas of March had already participated in at least one hackathon 
previously. The most common hackathons in which they participated were AT&T, Cal Hacks, and Ideas of March. 
This finding indicated that our students are looking for more opportunities to participate in this type of event and 
that students enjoy them and find them useful enough to participate in multiple events. 

The top two reasons for participating in the hackathon were: having the learning experience of being in this 
hackathon (65%) and learning Android Programming (52%). The third reason for their participation was to have the 
experience to work on an authentic project within a team (30%). Since a large percentage of students each year list 
learning Android Programming as a reason for participation we feel this is an encouraging sign that lack of prior 
experience is not seen as a barrier to participation. 

The top three things they liked the most were: having mentors (26%), the working environment and 
atmosphere (22%) and developing a product for an authentic client (17%). Some of the comments about the 
helpfulness of the mentors were: "Having the mentors around was immensely helpful, and being surrounded by 
dozens of people all working just as hard was a good motivator. " and "I liked the environment. In large 
hackathons, there is really no opportunity to receive one on one help from mentors. The mentors were extremely 
helpful. I also enjoyed the fact that we finished our app. " 

Regarding the perceived helpfulness of the mentorship, 94% of participants considered that the mentors 
were very helpful. Some of the adjectives mentioned in their feedback were “ awesome ”, “ patient ”, “ very good”, and 
“ supportive ”. The mean of a Likert scale measuring the helpfulness of the student mentors, was 4.74, where 1 was 
“Not helpful at all'’ and 5 was “Extremely helpful”. For professional mentors, the mean was 4.39, with no significant 
difference between these groups. Some of the comments about the mentors were: "There was always a mentor 
willing to help, and they were definitely a huge help with my team. Every’ single one of them was great. ” and "The 
mentors were amazing and I cannot express this enough. No matter what question we had the mentors, professional 
and student mentors knew exactly what we needed in order to help us out. " 

Mentors seemed to have been so helpful that when asked ways to improve the mentorship process, 54% of 
participants responded that they had none. Just 15% of participants mentioned that there could be a better way to 
reach mentors when needing them: since teams of participants were spread across the entire building, mentors 
walked around in regular intervals asking whether there were any questions. Participants had to walk to our 
headquarters classroom to locate a mentor if they had a question or issue that needed immediate attention. 

Since this event requires the collaboration of all team members, we also asked the students about their 
perceived experience working within a team. The majority (56%) reported having a very positive experience. As 
part of their open-ended feedback, they used adjectives such as “great”, “very good”, “awesome”, “fantastic” and 
“fun”; 22% of them reported having a positive experience (“positive”, “good”, “okay”). A few of them 4% reported 
it as not positive or challenging: "My experience on a team was challenging. I worked with people I hadn't worked 
with on a team before. I was able to learn more about how others work and how to approach them when they aren't 
communicating about their efforts on the project." Regarding their perceived level of contribution, 39% considered 
that there was an equal task division; 22% considered having contributed more than their teammates; and 13% 
considered having contributed less than the rest of the team. 

We were also interested in knowing whether the students had learned something from their participation; an 
open-ended question in the survey asked them to describe what they have learned. 48% mentioned they had learned 
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Android programming, which was one of the main reasons for having joined the hackathon. 39% reported having 
learned working within a team in an authentic project. Some of their comments were: "Teamwork is essential if one 
hopes to produce a great product within a short amount of time. " and "How to work with a team under a strict time 
constraint and a lot of pressure are two things I learned that I don 't think I could have learned in my regular 
classes. " 

When asked whether they had made new friends during the hackathon, 64% reported having done so and 
also strengthen existing relationships. Moreover, 50% of them reported that it was likely that they would continue to 
collaborate with the people they met in some other event: “No one knew each other and we got the opportunity’ to 
work with each other for the first time, it was a great experience and we were able to become friends at the end. I 
would be delighted to work with my again in the future if the opportunity exists. ” Not only did they socialize with 
their teammates but with other participants as well: “I knew the people who were on my team and we have been 
friends for a while. I did however make new friends on other teams. Every’one was very’ nice and supportive. ” 

Improvements Implemented 

In order to enhance the learning experience of students’ participation in the hackathon organized at our 
institution we implemented the following strategies: (1) offered workshops related to the technologies that will be 
used, prior the event; (2) actively recruited professional and student mentors to assist participants; (3) invited local 
non-profit organizations and local small businesses to pitch an idea that students could work on; (4) provided free t- 
shirts to increase the sense of community among participants. 

Conclusions 

We have offered the annual hackathon event since 2012 with increasing participation. Results from surveys 
indicated a very positive learning experience. In general, students have found it very valuable to have the 
opportunity to work on an authentic project as part of a team. In addition to improving their programming skills, 
participating in a hackathon also helps students work on their soft skills such as project and time management, 
interpersonal communication, motivational strategies, and dealing with unexpected issues. 

Overall, we feel that these results highlight several important things about the Ideas of March hackathon 
model. (1) Students have an overall positive experience despite the tight time constraints of the event; (2) Access to 
mentors helps students feel confident taking on an app even if they perceive that their Android programming skills 
are weaker; (3) Working with community partners to provide application ideas is an important part of the process 
that gives the students experience working with a real-world client and also provides motivation to produce 
something valuable to the community; (4) Student learning does take place during the hackathon and students are 
able to develop both technical and soft skills during the weekend; and (5) The experience of working with a team in 
a high pressure situation is a valuable part of the process. 
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Abstract 

Facilitating effective distance learning interaction depends on the use of current technologies. 
Substantiated technology integration strategies are needed for distance educators to make informed choices 
about technologies as well as learning goals. Technology attributes and pedagogical factors that contribute to 
distance learning interaction are discussed. The purpose of this work is to provide guidance for technology 
integration in order to enhance effective interaction in distance learning. An interaction framework prototype, 
arising out of a design and development research effort, is also presented. 
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Distance learning is an important and growing part of educational practice (Mclsaac & Gunawardena, 
2000). With the rapid development of emerging technologies, distance learning involves more options in course 
resources, diversity of learning activities, and types of communication tools for supporting teaching and learning 
(Beldarrain, 2006). More and more studies have placed increasing emphasis on distance learning. 

Based on transactional distance theory, the distance of importance in distance learning is not the 
geographical separation of learners and instructor, but rather the intellectual, social, and cultural distance 
(Kearsley & Moore, 1996; Moore, 1989). Adopting appropriate technologies and instructional strategies to 
foster interaction can minimize such psychological distance (Beldarrain, 2006). The rapid growth of more 
advanced technologies has created opportunities for more sophisticated interactions in the technology-mediated 
environment (Anderson, 2003) that can accelerate learning, our AECT conference theme. 

A challenge facing many educators is how to integrate technologies for the design of distant 
interactions that support learning goals. Pressed by a lack of time, resources, and instructional design expertise, 
many distance educators often go it alone in trying to make choices for interaction. Current distance learning 
practices may not maximize the interactive learning possibilities of today’s technologies. Technologies are often 
reduced to repositories for content (Richards, 2006), raising several, largely unanswered, questions for 
consideration by distance learning researchers and practitioners: Flow can distance educators integrate 
technology to foster interaction from a distance that supports learning goals? What pedagogies for integrating 
technologies encourage meaningful interaction? This paper provides guidance for technology integration to 
enhance effective distance learning interaction by providing a framework prototype that educators can use to 
make informed instructional design decisions. 

Review of Literature 

Historically, interaction mainly focused on learner-instructor interaction in the face-to-face classroom 
(Anderson, 2003). With technological developments, there is wide recognition that interaction can be supported 
by the use of technologies. Interaction can take diverse fonns such as synchronous interaction and asynchronous 
interaction (Kearsley, 1995). The concept of interaction has evolved from the dialogue between students and 
instructors in a traditional classroom-based setting and has been broadened to include technology-mediated 
interaction in distance learning (Anderson, 2003) among learners in addition to with an instructor. 

Considerations of Purposes of Interaction 

Wagner (1997) advocated that the concept of interaction should shift from learning agents to learning 
outcomes, especially in the information age, since such a shift could aid in employing instructional methods to 
improve learning performance. Therefore, instead of focusing on learning entities involved in the interaction 
process, some researchers define interaction in terms of purposes and functions. 

Different technologies support different modes of online interaction. Real-time technology and 
delayed-time technology afford differences in the interaction experience. Research indicates that delayed-time 
technology is more effective in facilitating task-oriented communication and reflective activities, whereas real- 
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time technology promotes more social interaction (Chou, 2002; Meyer, 2003). Social interaction may not 
directly contribute to the learning goals of instruction, but may foster a more positive learning atmosphere. 
Wagner (1997) suggested twelve types of interaction focused on learning outcomes: interaction for 
participation, communication, feedback, elaboration and retention, self-regulation, motivation, negotiation of 
understanding, team building, discovery, exploration, clarification of understanding and closure. Choosing 
technology according to specific interaction purposes is essential. 

Consideration of Technology Attributes 

Focusing on technology attributes rather than technologies themselves is a more productive way to 
consider instructional technology (Clark, 1983; Kozma, 1991). The attributes of a technology refer to its 
capabilities, such as the provision for active responding or the capability to provide auditory and visual channels 
simultaneously (Levin et al., 1990). Wagner (1994) distinguished interaction and interactivity. Interaction 
functions as the property of learning events, whereas interactivity refers to technological attributes (Anderson, 
2003). Roblyer and Wiencke (2003) indicated that the distinction between interaction and interactivity is 
important for researchers who investigate technological attributes to increase instructional interaction. Two-way 
communication, user control, real-time interactivity, and delayed-time interactivity are four key technological 
attributes that can foster interaction (McMillan & Flwang, 2002). 

Two-way communication is a key technological attribute to foster interaction (Northrup, 2002; Roblyer 
& Wiencke, 2003). McMillan and Hwang (2002) characterized two-way communication as a function of 
interpersonal interaction and the capability for providing feedback. Videoconferencing, teleconferencing and 
computer-mediated communication are two-way interactive technologies. 

Control is an important attribute concerning interaction between the learner and the content. With 
advanced technologies, more learner control opportunities are available including: the depth of study, sequence 
of instruction, pacing, and style of presentation (Gilbert, 1998; Wagner, 1994). The more a distance learner is 
offered options for meaningful control of the technological learning environment, the more likely that learner- 
content interaction will support positive learner experiences. 

Wagner (1994) concluded that real-time interactivity is “one significant attribute of the technologies 
used in current educational enterprises that sets them apart from previous technology” (p. 6). Real-time 
interactivity technology supports high social presence and immediate feedback (Chou, 2003). Research by Tu 
and Mclsaac (2002) indicates that social presence and immediate feedback positively influenced interaction. 
More social interaction can be supported by real-time interaction (Chou, 2002; Jonassen, 2001; Meyer, 2003). 
Real-time interaction is also effective in developing a sense of community among learners since it supports 
more cohesive interaction (Duemer, 2002; Jonassen, 2001). 

Research demonstrates that delayed-time technology, with its flexibility in terms of time and place, 
also fosters interaction (Chou, 2002; Kiousis, 2002; Vrasidas, & Zembylas, 2003). With delayed-time 
technology, more interaction opportunities are provided because learners can access and process interaction over 
longer periods of time and space. With more time for learners to compose responses in discussions, there is 
greater opportunity for reflection, a primary advantage of delayed-time technology. More time for interaction 
and the nature of written communication itself support interaction effectiveness in regards to reflection, critical 
thinking, and group problem solving (Chou, 2002; Jonassen, 2001). 

Methods 

This research project used a design and development research methodology, a pragmatic type of 
research, which is particularly relevant to the field of instructional design and technology (Richey & Klein, 

2007). This study was classified as Type 2, now known as model research, in that it addresses the design and 
development of a new framework with a generic focus (Richey & Klein, 2007). The following three phases were 
employed to develop a framework for designing interaction in distance learning: analysis, development and 
evaluation, and revision (Richey & Klein, 2007). 

Research Participants 

In this study, the framework was validated by a purposefully selected group of expert reviewers (Patton, 
2001). Three experts were selected based on their expertise and contributions related to distance education, 
learning theory, and instructional design. Prior to identifying participants, the researchers obtained approval for 
the study from the Institutional Review Board (IRB) at Virginia Tech. 

Data Sources and Data Analysis 

Phase one: Analysis. The first study phase was an analysis phase during which data from a systematic 
literature review were used to identify relevant elements of a distance learning interaction framework. The main 
sources of literature included academic journals, academic databases, online journals, books, and doctoral 
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dissertations published from 1995 to 2015. The selected literature met three inclusion criteria, content is: (1) 
peer-reviewed, (2) theoretical work or empirical research, and (3) situated within the context of education. 
During Phase I of this study, inductive content analysis was used to analyze the data (Erickson, 1986). 

Phase two: Development and evaluation. In the second phase, development and evaluation, the 
findings from the analysis phase were used to develop the framework. The resulting framework was reviewed 
by three experts for feedback. The online rubric for the expert review was custom design and administered 
through Qualtrics survey software. The rubric requested usability feedback on a number of framework elements 
including: relevance, guidance, detail, clarity, organization, structure, and format. 

Phase three: Revision. Revision was the third phase of the study. The three experts were contacted 
and their feedback was used to make improvements to the framework. Data collected from the three experts 
were coded and analyzed through a constant comparative analytic method (Rossman & Rallis, 2003). 

Developing a Framework 

Based on the findings from a systematic literature review of interaction research in distance learning, 
the framework regarding technology integration for purposeful interaction in distance learning was created (see 
Appendix A). The framework considers technology attributes, focusing on the level of interactivity offered by 
technologies. Technologies differ greatly in their potential to foster interaction in terms of level and functions 
(Barker, Frisbie, & Patrick, 1989; Bates, 1990; Fleeter, 2000; Roblyer & Wiencke, 2003; Wagner, 1994). The 
framework identifies key questions and components to be considered for educators to foster interaction through 
technology integration. Guidelines, along with supporting research, for each component are also provided. 

The framework outlines four key questions for designers to follow as they utilize technologies to 
enhance interaction in distance learning. These questions are: 

1 . Do available technologies support identified interaction purposes? 

2. Do available technologies support different types of interaction? 

3. Flas temporality of technologies been considered? 

4. Is the form of communication supported by the technologies considered? 

5. 

Expert Review 

Generally, the results of expert review indicated that the framework could be helpful for improving the 
quality of interaction design in distance learning experiences. Reviewers also pointed out that a systematic 
literature review served as a solid foundation to develop this theoretically- and empirically-grounded framework 
for guiding interaction design in distance learning. The third reviewer stated that, “I so appreciate that you build 
your case for your design on top of research, perspectives, opinion and evidence that already exist in the 
research literature to ensure that your design is predicated on reliable information.” This reviewer also stated 
that, “this is a nice enough overview of design considerations from the educational technology literature.” While 
the proposed framework was generally viewed to meet its intended use, experts offered several suggestions for 
improvement. Opportunities for improvement to the framework are discussed the below. 

Reviewer one suggested that social presence can be addressed more in the framework. Therefore, a 
short explanation about the relationship between social presence and interaction and an explanation of the term 
social presence were added. 

The feedback from reviewer one also raised another issue regarding implementing the technology 
integration. This reviewer explained that: 

Some instructors have very little control over the LMS and the tools they are given in it. In my case, 
there are many LMS options that I do not use and I want to turn them off, but Information Technology 
Services refuses to enable that level of instructor control over the LMS. Also, supplementing the LMS 
with additional technologies is a great idea, but policies on some campuses require instructors to only 
use the institutionally supported LMS, or to only include other technologies if they have been properly 
vetted by the (usually non-academic) campus Information Technology groups. I believe your guidance 
in this area should recognize those issues and soften a bit. 

This reviewer suggested that a related framework statement be reworded as, “When possible, supplement LMS 
with additional technologies that do support interaction purposes.” To address this suggestion, the text “when 
possible” was added in the first key question of the framework. 
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Reviewer three suggested including some contemporary technologies such as interactive web design, 
app design, mobile design, and games in example technologies in the guidelines. To address this suggestion, a 
statement was added after the table indicating that, since technologies will continue to change, the technology 
integration guidelines were limited intentionally to address only affordances and not specific technologies (that 
may or may not be current) and that the examples in the framework are not exhaustive but only representative. 
Appendix B presents the revised interaction framework. 

Discussion 

Interaction is an essential component contributing to the effectiveness of distance learning. Facilitating 
interaction in distance learning depends heavily on the use of technology. A valid challenge facing many 
educators and administrators in distance learning today is how to successfully incorporate technology into 
instruction to enhance and support interaction. Researchers have asserted that the current lack of guidance 
hinders improvement in the quality of interaction in distance learning (Roblyer, 2003; Wagner, 1994). In order 
to take advantages of interaction possibilities enabled by technologies, the investigation of technological 
attributes as well as the relationship between these capabilities and learning is necessary. However, research 
regarding technology integration for purposeful interaction in distance learning is still sparse. Studies in distance 
learning must call on a full range of learning and design theories to identify strategies that make the best use of 
technology to support interaction. 

Considering the findings, seeking additional insights into various contemporary or emerging interactive 
technologies and offering specific and detailed guidelines for different types of commonly used interactive 
technologies would strengthen the application of the framework. Currently, the framework provides technology 
integration guidance for fostering interaction in relatively general terms. 

Our next step is to invite more theoretical experts and distance learning practitioners to review this 
framework further. One reason to include distance learning practitioners, the target user audience, is to judge the 
real-world applicability of the framework. Further review can make the framework more robust for use in 
distance learning instructional design. 
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Vrasidas & Zembylas, 2003). 

Can provide an extended time period for interaction and more opportunity for reflection, critical thinking, and 
group problem solving (Angeli, 2003; Chou, 2002; Gilbert & Moore, 1998; Jonassen, 2001; Markus, 1994; 
Mclsaac & Gunawardena, 2000). 



supported by the technologies • One-way communication 

considered? ■ Two-way communication 

o Fosters interaction (McMillan & Fiwang, 2002; Northrup, 2002; Roblyer & Wiencke, 2003; Woo, 
2007). 

o Supports interpersonal interaction and feedback capabilities (McMillan & Fiwang, 2002; Oliver, 
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Appendix B 

A Revised Framework for Fostering Interaction In Distance Learning through Purposeful Technology Integration 

Key Question Guidance 

Do available technologies Analyze the need of interaction in terms of serving specific purposes in support of learning: 
support identified • Social interaction and instructional interaction (Berge, 1999; Gilbert & Moore, 1998). 
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technologies been • Supports high social presence and immediate feedback (Chou, 2003; Tu, 2001). 

considered? o Social presence primarily contributes to learners’ social emotion. 

o High social presence is more likely to result in more social interaction in distance learning (Swan, 2002). 

• Supports more social interaction (Chou, 2002; Jonassen, 2001; Meyer, 2003; Pena-Shaff, Martin & Gay, 2001). 

• Effective in developing a sense of community among learners since it supports more cohesive interaction (Duemer, 
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Collaborative Communications in the Classroom 


Patrice C. Nyatuame 

Morehead State University, 2015 
150 University Boulevard 
Morehead, Kentucky 4035 1 

Descriptors: communications, collaboration 

The overall goal of collaborative communication in the classroom is to look at how teachers relate their 
instructions to students, provide effective communication to administrators, co-workers and parents. There are many 
companies that claim that their tools are the best for collaborative communications. The focus is not to choose a 
particular vendor or to recommend a specific forms of communication, but to investigate better ways to 
communicate by collaborative means. 

The inspiration for this article was a symposium that took place in Lexington, Kentucky, sponsored by the 
Kentucky Department of Education, Division of Student Success. The symposium, entitled “Increasing Engagement 
to Close Achievement Gaps: Symposium 2015,” was organized to inform educators about digital learning concepts 
and introduce emerging technology. Teachers shared their ideas and presented best practices for their classroom. A 
common theme that derived from the symposium was finding a better way to communicate through technological 
means that would both better communications and successfully provide avenue for student achievement. 

Only 60 teachers out of thousands were able to attend the symposium because of limited space and money. 
Some educators admitted that this was their first conference and their first time experience with collaborative 
communication. Even though technology is present in their school and school district, they hesitate to use it because 
of lack of professional development and technological know-how. Teachers agreed that this conference brought new 
light to digital learning, but most apps that allowed for time saving tips and tricks that would be useful in their 
classroom. 

It is my hope that you the reader feel encouraged to try out apps that would improve instructional 
engagement and excite learners to take this dialog to a new expiring level. The goal is to introduce limitless 
exploration in a cloud-based world where collaborative tools and communication application suits your needs in the 
classroom. 


Is Collaborative Communications, a New Education & Discerning Curricula? 

Today, collaborative communications are imperative to becoming an efficacious teacher and learner. In 
order for teachers to connect students to real world tasks and link new information to prior knowledge, collaborative 
means are a necessity. Collaborative learning allows students an opportunity to communicate with the world and 
gives them huge advantages that might not be available through traditional means. In the traditional classroom, there 
is a one way flow of information, from the teacher to the students. 

Once upon a time collaboration was simple dialog and new collaboration was sending Microsoft Office 
Word documents by email to a co-worker and/or student. Manipulating the document, re-attaching the document and 
sending it on to the next participant. Microsoft Office Word and many other programs have come a long way since 
the age of attaching and re-attaching documents for others to input changes, access information, evaluate and/or 
track. With the complex problems that we face today with issues such as cyber-bullying, sexual harassment and 
outbreaks of diseases we are constantly investigating solutions that begs for emerging technology tools. 

Emerging Technologies in the Classroom 

Traditional teachers might focus on writing essays and chalkboard exercises, collaborative teachers use L 
earning Management Systems and interactive whiteboards for exercises. 

There are many learning and training software solutions such as Bride, Mindflash Online Training, 
SchoolKeep, Grovo, DigitalChalk, Litmous LMS, Administrate, TOPYX LMS, Training Management Software, 
Lesson, ly and the list goes on and on. Most software claim to maximize student and employee contribution and 
improve in real time; however, what’s important about LMS and online content platforms is that; it is online. It 
allows you, the user, to create online, acquire mobile access, integrate other applications, utilize management tools 
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and provide a collaborative communication platform without murdering trees and spending tons of money on ink for 
copying and writing. 

Collaboration does not mean spending money to effectively communicate with co-workers, parents and/or 
students. Products such as Flowboard and Google Drive (formerly Google Docs) allows students to create, publish 
and share work on their IPad. Google Drive (formerly Google Docs), Google+ Hangouts, Join Me, Open Class, 
Citrix Podio and ResearchGate are free apps and tools that can be used in a classroom setting. 

Flowboard is a free online application that allows collaboration with co-workers, parents and students. You 
can create documents and share in real time. Google+ Hangouts is a free tool that allows video and voice 
communication. Join Me is a free screen sharing tool that can accommodate up to 10 meeting participants and 
includes internet calling and chat. OpenClass is a free open-learning environment that allows the exchanging of 
content and integrates Google apps for education much like a LMS. ResearchGate is a free app for professors and 
students that is used primary for research projects. SocialFolders.me is basically a drop box that allows students to 
manage content across various social networks. (McCrea, 2013). 

There are some application that cost, but are provide by Schools and Districts such as Office 365 which 
should not to be confused with Office Online. Let’s look at what’s free, what’s not and using what you got. What’s 
free is Office Online, in this package comes Word, Excel, PowerPoint and OneNote. Office Online apps are 
available on your mobile device, but note there are limitations as far as functionality. For example. Office Online 
does not give you the functionality of mail merge and so forth. If you look at the ribbon on Office Online and Word 
2013 they look almost identical; however, you might not see format painter and fonts such as text effects, 
typography or change case (Bradley, 2015). 

Office 365 asserts that one can store, sync and share files simply by using their apps. By using OneDrive 
for Business, Microsoft claims that on your 1 TB of storage you can securely store all your files and share them with 
co-workers, sync files across devices and have access to your files anywhere even when you are offline. 

Google also claims that they can make your collaborative communication easier. Google asserts that their 
tools allow one to create online documents, presentations and spreadsheets, without an office suite since everything 
happens in your browser via mobile app for IOS or Android. 

Rather you are an Office 365 or Google user you have the opportunity to share content by sending a link. 
The real point is that you can create documents and share them in real time. There are some free applications that 
will suit your needs as well as some application that you might have to purchase to suit your needs. (Tinzmann, 
Fennimore, Bakker, Fine, & Pierce, 1990). According to conversations at the Increasing Engagement to Close 
Achievement Gaps 2015 symposium, teachers exclaimed that professional development on collaborative 
communications in the classroom would enhance curriculum innovation as well as the role of learning and 
schooling. 
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Abstract 

The face of classrooms in the United States is shifting to reflect the needs of a changing society. Hence, 
online learning is prevalent. The number of students taking at least one online course has surpassed 7.1 million and 
represents 33% (21.3 million) of all higher education students (Allen & Seaman, 2013). With the growing number of 
online courses, credibility may be questioned. Are there effective evaluation processes in place? This article will 
describe the results of a mixed-methods study that identified students’ perceptions of the communication processes 
utilized in the presentation, delivery, and return of online evaluations of teacher performance. 

Key Words: online learning, student evaluations, reflective practice 

The face of classrooms in the United States is shifting to reflect the needs of a changing society; hence, 
online learning is prevalent. This growing market has demonstrated sustainability and credibility with higher 
education and the general public. Millions of students are engaged in online education encompassing $25 billion of 
annual tuition revenue (Gallagher & LaBrie, 2012). The number of students taking at least one online course has 
now surpassed 7.1 million and represents 33% (21.3 million) of all higher education students (Allen & Seaman, 
2013). In the 2013 Survey of Online Learning Report, “The percent of academic leaders rating the learning 
outcomes in online education as the same or superior to those in face-to-face instruction grew from 57% in 2003 to 
74% in 2013.” These numbers represent the current trends in education and offer insight into the wave of future 
students entering their collegiate careers. With the demand of online classes increasing, questions may arise as to 
the credibility of these courses, as well as to the instructional quality of those teaching these online courses. 

Students’ evaluations of online course content and faculty instruction are important. Liu (2012) advised 
that student evaluations of online learning are critical to establish and maintain the integrity of online learning. 
Additionally, student evaluations of teaching have been a source of credibility with regard to advancement for 
teachers along with their continued reflection and growth as educators. In this article, the results of a 1.5 year-long 
research study about student response rates for evaluations of faculty performance in higher education online classes 
will be described. A review of the current literature regarding student evaluations of teaching in non-traditional 
online classes will be presented, as well. 

Purpose 

The purpose of this mixed-methods study was to identify the faculty and students’ perceptions of the 
communication processes utilized in the presentation, delivery, and return of online evaluations of teacher 
performance. Norris and Conn (2005) expressed the need to collect impressions of student insight that would allow 
identification of variables that encourage or distract from their responses to online-delivered evaluations. Berk 
(2013) noted that despite the numerous research articles written on student evaluations of teaching in online 
environments, few have evaluated non-traditional online and/or blended classrooms. As a preliminary examination, 
faculty and students were asked to engage in a survey to gain understanding into the current use and perceptions of 
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communication strategies used in online classrooms to deliver online evaluations. Additionally, interviews and a 
thorough literature review were conducted to answer the questions of this study. 

Hypotheses 

Student evaluations are important for online courses. When a concerted effort is given by faculty in higher 
education to explain the purpose of student evaluations, students will complete the evaluations and provide feedback 
to improve teaching and learning. 


Research Questions 

1 . What communication processes in the presentation and delivery of online student evaluations does faculty 
engage to encourage the completion of these evaluations? 

2. How do students perceive and respond to communication regarding the completion and importance of 
online evaluations? 


Significance of the Study 

This study is significant because it furthers the investigation into methods and strategies that may enable 
students to complete online evaluations in higher education settings. A high student response rate is significant to 
faculty in higher education, as contract renewals are often dependent upon these student evaluations. Additionally, 
tenure and promotion committees highly value student evaluations. Berk (2013) surmised that the format of online 
teaching varies greatly from the online classroom indicating that research must begin to address the needs of the 
entire evaluation process specifically for online and/or blended courses. To date and per the researchers’ review, no 
studies have focused specifically on students’ evaluations of online teaching in this specific context. 

Literature Review 

The research in the area of online student evaluations of teaching in traditional teaching environments has 
produced varied levels of concern with regards to response rates (Adams & Umbach, 2012; Anderson, Cain, & Bird, 
2005; Guder & Malliaris, 2010). Guder and Malliaris (2010) indicated a dramatic drop in response rates for student 
evaluation of teaching in transition from paper-pencil to online formats, -25.99% from the years of 2005-2008. This 
lower response rate within online evaluation is reflected greatly in the research (Berk, 2012; Anderson et ah, 2005; 
Opengart & Mau, 2012; Morrison, 2011; Nowell, Gale, & Handley, 2010; Nair, Adams, & Mertova, 2008; Stowell, 
Addison, & Smith, 2012). Several studies suggest that students who perform better academically are more likely to 
respond to student evaluations (Adams & Umbach, 2012; Anderson, Cain, & Bird, 2005). Adams and Umbach 
(2012) indicated that students who showed little investment in the class viewed the evaluation as a reminder of their 
substandard performance, thus influencing their lack of participation. They also reported that students are more 
likely to respond to evaluations within their major study focus. This sense of responsibility is credited to the major 
of focus, indicating a feeling of a sense of duty to the department. 

Online evaluations have been shown to produce more thoughtful and lengthier comments from which to 
guide practice and pedagogy (Anderson, Cain, & Bird, 2005; Guder & Malliaris, 2013). Guder and Malliaris (2010) 
reported benefits in the transition to online formats, as the comments made by students in the online evaluations 
increased by 19% of students versus the paper-pencil evaluations. In addition, a reported 149% increase of average 
number of words written in the comments section was also observed. Students reported to prefer to complete 
evaluations outside of class time so that they can produce more “effective” and “constructive” (p. 40) responses. 
Online versus traditional paper-pencil formats have also shown a decrease in time requirements from multiple 
perspectives. Anderson, Cain, and Bird (2005) reported that students spent ten minutes or less on online student 
evaluations of teaching and 25 minutes on paper-pencil. Additionally, staff workload decreased by 30 hours when 
evaluations were completed online (Anderson, Cain, & Bird, 2005). Gravestok et al. (2008) indicated that this 
decrease in staff time results from reducing the need for in-class distribution, collection, scanning, and storing data. 

While there are researched advantages and disadvantages to the use of online student evaluations of 
teaching, research focus has moved to the ability to increase the response rates of students in the online formats. 
Overall, online student evaluations of teaching produce lower response rates than traditional in class-methods 
(Stowell, Addison, & Smith, 2012); however, several studies have indicated that response rates can be increased 
through the use of multiple and varied strategies. In particular, penalty and reward strategies offer an opportunity to 
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increase student response rates for online delivered student evaluations of teaching. Ravenscroft and Enyeart (2009) 
supported that including the evaluation as an assignment has shown to produce desired results for response rates. 
Anderson, Cain, and Bird (2005) reported on the transition of the University of Florida, School of Pharmacy, and 
their culture-wide adoption of the required completion of online student evaluations of teaching to obtain a grade, 
ultimately with favorable results. In addition. Berk (2013) identified positive incentives such as extra credit or 
dropping the lowest score likewise produced positive results for response rates in online delivered student 
evaluations of teaching (Dommeyer et al., 2004; Johnson, 2003; Pmnty, 2011). 

While reward and penalty incentives have shown to increase response rates, Norris and Conn (2005) 
advised that “simple” communication strategies can show an effect in the positive response rates of students without 
withholding grades or the penalizing of students. Norris and Conn compared the paper-pencil versus the online 
evaluation systems and found that while response rates for online evaluations were generally lower than traditional 
paper-pencil evaluations, response rates were significantly higher for teachers who communicated with their 
students. Faculty who provided students information about the importance of the online evaluations, instructions on 
how to complete them, and reminders to do so were successful. 

Norris and Conn (2005) completed a series of studies to evaluate a baseline and communication strategies 
for response rates for online evaluations. In the evaluation, the researchers discovered that through the use of these 
simple communication strategies regarding the importance of student evaluations of teaching, students were 
significantly more likely to complete online evaluations (a rate of 74%) than those students who did not receive 
communication strategies (baseline rate of 34%). This focus on the communication regarding the delivery and 
importance of online student evaluations of teaching was evident more recently through the work of Knight, Naidu, 
and Kinash, (2012). In their research, the indicated that “Students were more likely to participate in evaluation 
surveys if they felt that their feedback made a meaningful contribution.” (p. 226). Through the collaboration of 
students in the process of development and communication of student evaluations of teaching, the university boasted 
an 89% response rate for online evaluations. The authors reported that student engagement and collaboration was 
the key ingredient in the “achieved response rates that exceed those presented elsewhere in the literature.” 

Theoretical Framework 

Reflective thinking is the theoretical framework for this study. Based on John Dewey’s work, reflective 
thinking is “an active, persistent, and careful consideration of any belief or supposed form of knowledge that 
supports it and the further conclusions to which it tends” (Dewey, 1933, p. 6.) Learning and performance 
improvement outcomes can result with the implementation of reflective practices (Handal, Vaage, Carlgren, 1995). 
The significance of student evaluations of teacher performance in higher education cannot be understated. 

Method 

Participants. Two categories of participants, students and faculty, were eligible for this mixed-methods 
study. Ages ranged from 18 to 67 years of age. Participants were from a university in Western Pennsylvania who 
were involved in one or more online classes. 

Materials. With permission, the survey tools used in this study were adapted from the original 2005 
surveys used by researchers, Norris and Conn. Three local experts in research theory and design were asked to 
review and analyze the research questions and the survey tool to confirm valid and reliable information for this 
study. The instrument was revised accordingly with minor adjustments being made prior to the survey distribution. 
(See Appendices A and B). 

Procedures. After receiving the institution’s approval to conduct this study from the Institutional Review 
Board (IRB), names and e-mail addresses were obtained through the university’s Applied Research Lab (ARL) of 
those faculty and students who taught or took an online class. The information was coded to maintain 
confidentiality. Via an online survey, the researchers distributed this tool to students (n=168) and faculty (n=9). 
Following the survey, those faculty and students who agreed to talk further were asked follow-up questions. 

Analysis and Findings 

To address the research questions, surveys were sent to students and faculty involved in online classes. 
Initially, the response rate was low; therefore, the surveys were re-distributed. In the end, 25 students and 5 faculty 
members responded to the online survey; thereby, confirming a response rate of 17%. Additionally, five students 
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and five faculty volunteered to be interviewed in order to thoroughly address the issues of this study. Thirdly, a 
thorough review of the literature was conducted and trends were compared to the results of this study. 

Demographic information collected indicated that of the students who participated 64% were between the 
ages of 18-27 (n=16), 15% were between the ages of 28-37 (n=4), 8% were between the ages of 38-47 (n=2) and 
12% were between the ages of 48-57 (n=3). Twenty-eight percent (28%) of the participants were male (n=7) and 
72% were female (n=18). Predominately juniors and seniors completed the survey. Of the respondents, 68% were 
juniors (n= 17) and 32% were seniors (n=8). Of the three female and two male faculty respondents, one was 
between the ages of 38-47; three were between the ages of 48-57; and, one was between the ages of 58-67. 

Research Question No. 1: What communication processes in the presentation and delivery of online 
student evaluations does faculty engage to encourage the completion of these evaluations? In response to the 
question, “When was the end-of-the-semester course evaluation first announced to you?” responses varied. Twenty- 
eight percent (n=7) indicated that course evaluations were announced “A week before the last day of the semester.” 
Twenty-eight percent (n=7) indicated that the course evaluations were announced “two weeks before the last day of 
the semester,” and 28% (n=7) indicated that course evaluations were announced “More than three weeks before the 
last day of the semester.” Sixteen percent (n=4) selected the “other” category and indicated that the course 
evaluations were not announced. Faculty responses varied. Two reported “A week before the last day of the 
semester;” two noted “three weeks before the last day of the semester;” and, one marked “more than three weeks 
before the last day of the semester.” The student responses to “How was the end of semester course evaluation first 
announced to you?” indicated that 60% (n=15) stated that course evaluations were announced via email; 20% (n=5) 
announced it through class discussion; one response indicated it was posted to the website; and, 16% (n-4) said it 
was never announced. One hundred percent of the faculty who responded to the online survey said that they posted 
it on the website and discussed it in class. 

Follow-up interviews were conducted with five students and five faculty members who were involved in 
online courses. All five of the interviewed faculty members stated that they posted a notice of the student 
evaluations on their websites. However, only two of the five students mentioned that they had seen these notices. 
When asked, “Do you do anything other than posting in the websites?” four of the five participants responded, “No.” 
Student interviews revealed similar results. One faculty member said that she had shared with her students the fact 
that “she was already tenured and that she just wanted to make her class better.” However, her plea was ineffective, 
as her response rate that semester was simply 1 1% (2 of 18 students). She was somewhat surprised because she had 
received numerous positive e-mails from students throughout the semester and she was hopeful that her evaluations 
would be reflective of those messages. Another faculty member stated that his student evaluation response rate “was 
weak.” “I sent out several pleas for participation, but no incentives. It did not seem to matter.” 

Research Question No. 2: How do students perceive and respond to communication regarding the 
completion and importance of online evaluations? 

With regard to the importance of faculty evaluations and the weight they carry in higher education, results 
produced interesting outcomes. To the question, “Did the instructor discuss the value of end-of-semester course 
evaluations with you?” student and faculty respondents reported strong indication that they were not. Ninety-six 
percent of the students (n=24) responded “No,” and 100% of the faculty reported the same. In response to the 
identification of reminder strategies utilized by faculty, students overwhelmingly reported that no reminder 
strategies were used (100%). Faculty participants also confirmed that no reminder strategies were used. Negative 
reinforcement strategies, such as withholding grades or access to final grades, was explored through the question of 
“Did the instructor require you to complete the evaluation (e.g. in order to access the last assignment)?” One 
hundred percent (100%) of the students and faculty responded “No,” indicating that negative strategies were not 
utilized in the process of course evaluation. 

During the interview sessions, one faculty member stated that she gave two bonus points based on the 
honor system when students completed the evaluations. She said, “Students wrote their names under the discussion 
board and attested to the fact that they completed the student evaluations. After doing so, these students were 
awarded the bonus points.” The other four faculty members did not offer incentives of any kind. None of the 
students interviewed stated that they were offered incentives. During one of the student interviews, it was stated, 
“Since I never got anything out of it, why bother?” 

Based on the survey data, additional qualitative examinations were recorded in response to strategies used 
by professors and students’ perceptions on how to increase participation in online evaluations. To the survey 
question, “Please comment on any additional strategies used by the instructor to encourage you to respond to online 
course evaluations and please add any other insights into this issue (What would it take for you to complete the 
online student evaluation, if you didn ’t already do so?),” student responses conveyed both frustration and 
consideration into this issue. Frustration was expressed through means of the evaluation seeming to possess little 
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value with regard to the progression of teacher quality. In addition, students indicated that faculty members were 
ambivalent with regard to the completion of evaluations. As indicated in the non-existence of reminder strategies 
utilized, students expressed frustration with only one email with multi-level directions to access the survey. 

The method of accessing the student evaluations was cumbersome, and it required students to follow 
challenging directions. This sentiment was also revealed within a student interview, “The student evaluation system 
is complicated. You have to sign into your account, go to a section and scroll down, select this, and then click on 
this, and then, maybe, you can find the survey. It isn’t hard, but it takes patience and time.” 

Upon review of the literature and comparison to the analysis of this study’s results, discrepancies and 
similarities exist. The literature is very clear about there being a concern for student privacy; however, interestingly, 
privacy was not mentioned as a concern for the students who completed this particular survey. Additionally, Norris 
and Conn (2005) indicated that through the use of an evaluation protocol timeline with regard to the delivery of 
evaluations (two weeks prior to the end of the semester) and reminders (one week prior to the end of class with a 
completion date), response rates increased from 42% to 74%. No consistent timeline notice prevailed within this 
study. Furthermore, all student respondents indicated that no reminder strategies were utilized. The results of this 
study were similar in that response rates in this study were minimal which is consistent with the literature. 
Additionally, a minority of faculty members expressed the importance of these student evaluations to their students. 

Recommendations for Future Research 

Important insights were gained from this study; however, recommendations for future research exist. 
Indications that survey fatigue may play a role in the response rates of evaluations can be viewed from the 
participation rate of students and faculty members at this particular university. Adams and Umbach (2012) indicated 
that a threshold of survey participation is capped at 1 1 surveys. Further exploration of this phenomenon is 
warranted. 

While research appears to suggest that a protocol for evaluation delivery has the potential for increasing the 
response rates of online faculty evaluations, the translation to practice appears to require further investigation. 

Despite the numerous studies that indicate the need for communication of importance with regard to faculty 
evaluations, a continued effort to ensure that translation from research to practice is necessary. 

Limitations 

Limitations to this study included that it was administered to one university setting and, therefore, no 
generalizations can be made. However, the results are certainly consistent with the literature reviewed. 

Conclusion 

Most likely, online learning and teaching opportunities will continue to grow, and the need for reflective 
thinking and quality instruction will remain important. As found in this study, students perceive evaluations of 
faculty as a formality with little influence on the delivery of instruction. While universities utilize evaluations for 
important decisions regarding tenure and promotion, translation of this importance to students is essential. As 
indicated through this study, communication efforts of evaluation importance and reminder strategies by the faculty 
appears to be lacking in practice as perceived by students. While this study was concerned with the communication 
practices of one University’s faculty with regard to student evaluations of teaching in online classes, it appears that 
faculty communication requires further investigation in the student evaluations of online teaching. As this study 
indicated, communication practices were not a formal process in regards to delivery time of online evaluations, 
reminder strategies, and communication of importance to students; therefore, students’ perceptions of evaluations 
appeared to reflect the student evaluation of teaching as a formality with disregard for the evaluation importance. 
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APPENDIX A - Qualtrics Survey for Faculty 

YOUR WILLINGNESS TO PARTICIPATE IS INDICATED BY COMPLETING THIS SURVEY. 


Section 1 

Age Range: (please check where appropriate) 


Gender: (please check) 

Experience: 


18-27 [ ] 

28-37 [ ] 

38-47 [ ] 

48-57 [ ] 

58-67 [ ] 

68 + [ ] 

Male [ ] 

Female [ 1 


(B) Please select your faculty status (please check one) Assistant Professor | | 

Associate Professor i ] 

Professor [ ] 

Temporary j I 


Section 2 

1. When was the end of the semester course evaluation first announced to students (choose all that apply)? 

a. A week before the last day of the semester 

b. 2 weeks before the last day of the semester 

c. 3 weeks before the last day of the semester 

d. More than 3 weeks before the last day of the semester 

e. Other (please specify) 


2. How was the end of semester course evaluation first announced to the students (choose all that apply)? 

a. E-mail announcement 

b. Posted to class website 

c. Posted to class assignments 

d. Posted to a class discussion board 

e. Other (Please specify) 


3. Did you discuss the value of end of semester course evaluations with your students? 

a. Yes 

b. No 

If yes, please explain when and how: 
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4. Did you use any reminder strategies to encourage students to respond to the evaluation (choose all that 
apply)? 

a. E-mail reminder 

b. Posted reminder to class website 

c. Posted to class discussion board 

d. No reminder strategy used 

e. Other (please specify) 


5. Did you offer students credit (participation points, extra points, etc.) for completing the evaluation? 

a. Yes 

b. No 

If yes, please explain what kind of credit was offered: 


6. Did you require students to complete the evaluation (e.g. in order to access the last assignment)? 
If yes, please describe the requirement: 


7. Please comment on any additional strategies used for encouraging student responses to online course 
evaluations and please add any other insights into this issue. 


This adapted from the original rcport- 

Norris, J., & Conn, C. (2005). Investigating strategies for increasing student response rates to online- 
delivered course evaluations. Quarterly Review of Distance Education, 6(1), 13-86. 
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APPENDIX B - Qualtrics Survey for Students 

YOUR WILLINGNESS TO PARTICIPATE IS INDICATED BY COMPLETING THIS SURVEY. 


Section 1 

Age Range: (please check where appropriate) 


Gender: (please check) 

Experience: 


18-27 [ ] 

28-37 [ ] 

38-47 [ ] 

48-57 [ ] 

58-67 [ ] 

68 + [ ] 

Male [ ] 

Female [ 1 


(B) Please select your undergraduate status (please check one) Freshman [ j 

Sophomore [ ] 

Junior [ ] 

Senior [ ] 


Section 2 


1. When was the end of the semester course evaluation first announced to you (choose all that apply)? 

a. A week before the last day of the semester 

b. 2 weeks before the last day of the semester 

c. 3 weeks before the last day of the semester 

d. More than 3 weeks before the last day of the semester 

e. Other (please specify) 


2. Flow was the end of semester course evaluation first announced to you (choose all that apply)? 

a. E-mail announcement 

b. Posted to class website 

c. Posted to class assignments 

d. Posted to a class discussion board 

e. Other (Please specify) 


3. Did the instructor discuss the value of end-of-semester course evaluations with you? 

a. Yes 

b. No 

If yes, please explain when and how: 
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4. Did the instructor use any reminder strategies to encourage you to respond to the evaluation (choose all that 
apply)? 

a. E-mail reminder 

b. Posted reminder to class website 

c. Posted to class discussion board 

d. No reminder strategy used 

e. Other (please specify) 


5. Did the instructor offer you credit (participation points, extra points, etc.) for completing the evaluation? 

a. Yes 

b. No 

If yes, please explain what kind of credit was offered: 


6. Did the instructor require you to complete the evaluation (e.g. in order to access the last assignment)? 
If yes, please describe the requirement: 


7. Please comment on any additional strategies used by the instructor to encourage you to respond to online 
course evaluations and please add any other insights into this issue (What would it take for you to complete 
the online student evaluation, if you don’t already do so?) 


This adapted from the original report- 

Norris, J., & Conn, C. (2005). Investigating strategies for increasing student response rates to online- 
delivered course evaluations. Quarterly Review of Distance Education, 6(1), 13-86. 
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Creating an effective instruction design for an online course requires an analysis of the feedback loops 
embedded throughout the course and the foundational element of habitus. Current literature acknowledges the 
framework of the instructional design as a key necessity for online success (Merrill, 2006). Feedback loops grab 
data at different intervals of the learning process within the instructional design. Each loop returns information on 
the interactions and knowledge-building relationship between the student and the content, the student and the 
instructor, and the student and his peers. Feedback loops enhance teaching and learning, provide motivation and 
coaching, inspires hope and confidence (Sadler, 1998), enhance a learner’s self-evaluative capabilities (Price, 
Flandley & Miller, 2011) and are considered one of the most powerful influences on how people learn (Flattie, 

2012). Like a roadmap for the learner and teacher, feedback loops provide evidence, relevance, consequence, and 
action (Goetz, 2011). Undeipinning the instructional design is the theory of habitus, which impacts an individual's 
propensity for learning institutional and organizational designs for learning and teaching. 

Instructional experiences which make the acquisition of knowledge and skill more efficient, effective, and 
appealing. Merrill, Drake, Lacy, and Pratt (1966) affirmed that instructional design creates learning experiences and 
learning environments. Merrill’s instruction design components involved teaching and learning instmctionally 
designed around a problem-centered, real-world task. Merrrill’s (2006) five principles of instruction include the 
principles of demonstration, application, activation, integration, and task-centered engagement. 

New technologies bring additional challenges to embedding feedback loops into the instructional design. 
Text, audio, video, images, as well as augmented reality (AR) and virtual realities (VR), must each include a 
feedback loop from the learning and the teaching environments. While the ASSURE model (Smaldino, Lowthere, 

& Russell, 2008), ADDIE instructional design model and other instructional design models include assessment, 
evaluation, and the inclusion of feedback, they do not specifically look at the inclusion of feedback within 
individual components of feedback related to new technologies. AR handheld displays and world-fixed displays 
allow the learner to keep one foot in the real world while venturing two steps into an augmented virtual world. 

While the traditional classroom and the instructor may be present, the capturing of feedback within the augmented, 
virtual world is problematic unless the instructional design has allotments for measuring this feedback from the 
student within these environments. VR is even more problematic as the learner now has both feet (and often hands) 
immersed within a virtual world; as the learner maneuvers this learning environment, he is at the mercy of the 
software design and not within earshot of the instructor or the traditional learning environment. New instructional 
design models that employ a wide variety of relevant feedback loops, visual, auditory, social, and academic need to 
be re-examined. Dregg, Grover, & Kacierek (2010) contend, “Faculty teaching practices must be carefully 
evaluated in order to ascertain if faculty are creating an online learning environment that promotes interaction, 
communicating effectively with students, and providing adequate and timely feedback. 

Internal and external feedback loops within a digital setting need to be established to allow for interaction 
and ultimately academic success. Whether the feedback is ensconced in the pedagogical methods and strategies of 
the instructor or is embedded within the instructional design of the course, it functions as a necessary 
communication component. When the feedback is looped and measured, it becomes a mechanism that can assess 
student achievement, engagement, and motivation. Feedback loops force all parties— the learner, the instructor, and 
ultimately the institution— to analyze and evaluate its purpose in the symbiotic educational environment. When each 
loop communicates the needs of the learner and the instructor, it creates a mutually beneficial tool garnering data to 
implement future change within the classroom practices of the instructor and the instructional design of the course. 
Each feedback loop lends itself to the essential understanding of course goals and student insights by creating 
communication channels between the instructor and the content, the instructor and the students, the students to their 
peers, and the students to the content, and finally, the course to the institution. Feedback loops foster interactions 
between students, instructors, and peers, and each with content (Moore, 1989). The course is a feedback loop 
because it answers the question, “What am I supposed to do with all of this information?” The result should be that 
the course leads students to action. 
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As blended and online learning can pose difficulties for learners, the feedback loop captures the strengths 
and deficiencies within the course, and seeks to offer suggestions as to how students might learn to adjust to using 
new media under different paradigms and how instructors must adapt pedagogies, methods, and attitudes to the new 
media, recognizing that the new media is a possible vehicle for improving teaching and learning. As instructors 
adjust, they must also fashion their courses with viable feedback loops to encourage a dialogic process that does not 
solely respond to a conflict or disruption in the educational process but rather but focuses on continuous 
improvement. 

Learners advance along proposed academic paths through feedback. In a learning context, feedback can be 
used to improve the educational output of teaching and learning. In educational settings, feedback occurs from 
instructors, mentors, and peers. This can occur formally, such as formative or summative assessments, or informally 
during conversations or other similar means. Getzlaf, Perry, Toffner, Lamarche, and Edwards (2009) define 
feedback “as information provided from instructors to students about course activities in which students were 
engaged, including written assignments, conference postings and course interactions (p. 3). 

Feedback is just not a technical process but it serves as communication to the learner, it has motivational 
and coaching value, and it inspires hope and confidence in the student (Sadler, 1998). Bain and Swan (2011) 
summarize research about feedback and note the qualities of effective feedback that include feedback from multiple 
sources, including peers, timely, accurate, positive, specific, corrective, focused and concrete, This is consistent with 
guidance for practitioners that feedback to students is “descriptive, constructive, frequent, and timely" (Chappuis, 
Stiggins, Arter & Chappuis, 2006, p. 124.) 

Schimmel (1988) categories feedback into four separate areas: confirmation feedback, correct response 
feedback, explanatory feedback, and bug-related feedback. In confirmation feedback, the instructor simply confirms 
the answer is correct or incorrect. In confirmation feedback, students are presented the correct feedback whether the 
student provided a correct or incorrect answer. In explanatory feedback, the instructor provides specific guidance to 
help the student’s thinking move toward the desired answer. In bug-related feedback, the instructor provides 
feedback to correct the student’s misunderstanding of the learning concept. Like all other social interactions, 
feedback does not occur in distinct categories and frequently Schimmel’s categories are mixed in a single 
communication. 

Without modification from feedback, teaching and learning occur on unchanging tracks. Action occurs in 
the direction guided by the feedback; this, feedback becomes iterative as the learners understanding moves closer to 
desired outcomes. Like the feedback in an electrical amplifier, feedback becomes a loop. Information is returned 
into the process for change toward desired learning goals or output expectations. Goetz (2011) describes the 
components of a feedback loop: evidence, relevance, consequence, action. In the evidence stage in learning, 
students demonstrate their learning through artifacts (e.g., verbal responses, board work, written work, etc.). The 
instructor reviews the students’ work, immediately or during a grading cycle and responds to the students in a 
meaningful fashion to them: relevance. Further, the instructor’s guidance and direction “illuminate one or more 
paths ahead” (para. 8). Finally, the students must act to make changes in their understandings. The loop begins with 
students providing evidence of their learning. Like in electronics, the impact is an amplified student learning. For 
feedback to be effective in an educational setting, it must bring action, the feedback loop. 

An individual's propensity for learning is sculptured by the theory of habitus, which is more readily 
explained as ‘“the way society becomes deposited in persons in the form of lasting dispositions, or trained 
capacities and structured propensities to think, feel and act in determinant ways, which then guide them’ (Wacquant 
2005, as cited in Navarro 2006, p. 316). Yet, when the already sculptured individual enters a classroom, he enters 
into another habitus, and in higher education, this habitus is generally one in which he has chosen to engage and 
learn. Because of this, his predilections can often become as malleable as pliant clay through his eagerness to learn. 
As a result, even though the individual carries with him a personal habitus of “deeply ingrained habits, skills, and 
dispositions that we possess due to our life experiences,” he opens his mind up to the concept gaining the cultural 
capital of that organization’s institutional habitus (Bourdieu, 2005). 

To better visually this environment, one needs to examine the theory of habitus as it can apply to virtual 
learning spaces when knowledge is exchanged in an academic nature. Mauss (1934) defined the theory of habitus as 
those aspects of culture that are anchored in the body or daily practices of individuals, groups, societies, and nations. 
Another assertion of habitus views each learning environment as consisting of the set of dispositions, which result 
in practices, and representations that an individual can adapted to his environment with bias (Bourdieu, 

1990). Combining what Mauss and Bourdieu assert translates to cultural practices (such as the inclusion of core 
values) which expedite, but do not compromise, the end result of the teaching experience. In addition, Bourdieu 
further identified the theory of habitus as the system transposable dispositions and principles, which generate and 
organize practices necessary by a group. 
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To the 21 st century online academic community, the habitus of the learning environment can be used to 
explain the hidden agenda and the unique standards of core values, which can build the framework for academic 
interactions and knowledge exchanges. Within each blended or online environment, which is governed by the 
theory of habitus (Luke 2005), lies a hidden agenda. Hidden curricular issues are practices that may or may not 
impact the learning environment and ultimately student achievement (Jackson, 1968; Luke, 2005; Snyder; 1970). 
Freishtat and Sandlin (2010) contend that social media interactions are knowledge and information sharing 
discourses. Recognizing that each “socially oriented digital media produces a habitus within digital spaces” 
(Freishtat and Sandlin, 2010, p. 505), they claim it is critical to analyze not only the flows of information within 
cyberspace from the technologically mediated spaces (such as the social media or academic discussion forums) but 
also the social manner in which generations are conditioned to engage with others using the information they have 
received online. Because virtual communications have dramatically changed our communications and social 
interactions as well as a substantive effect on how people in our society behave and interact (Scheuermann & 

Taylor, 1997), the need to set the parameters for a hidden agenda are necessitated. When individuals enter into an 
online academic environment in which the hidden agenda of conventional core values has not been established, the 
strongest voice creates its own set of values and manners. When individuals enter into an online academic 
environment in which the hidden agenda of conventional core values has been taught, reviewed, and demonstrated, 
the institution assumes the dominant role and sets expectations for the course. 

When it comes higher education learning, “current literature suggests that there is a wide range of 
interacting personal and social attributes, as well as institutional practices, which impact on both performance and 
retention rates (Thomas, 2002, p. 426). Specifically, some of these factors can be identified as a learner’s personal 
habitus and the institution’s organizational habitus and institutional habitus. 

When it comes to online academic learning in higher education, one of the most influential personal and 
social factors influencing the non-traditional learner’s learning experience and his ultimate retention include centers 
on the individual’s habitus. Personal habitus, which is acquired through one’s familial or early academic training, 
can help maneuver or hinder one’s educational experiences. Because an individual’s habitus formulates the 
foundation of socialization skills and value-centered expectations that an individual uses to make their subsequent 
choices in life’ (Robbins, 1993, p. 159), he faces adaptation into a new habitus (specifically, institutional habitus), or 
he faces retaining his own personal habitus. While, other personal and social factors influence the non-traditional 
learner include academic preparedness, academic experience, financial and employment considerations, and family 
support and commitments (Thomas, 2002), recognition of one’s habitus becomes akin to recognizing and supporting 
diversity and social identity. Social identity usually refers to the processes of interaction by which one identifies 
‘others’ and is also identified by them; these processes then become the basis of self-identification as well (Kuhl, 
2000 ). 

On an organizational level, such as within an institution of higher learning, organizational habitus can have 
a major impact on a non-traditional learner’s learning experience and his retention. Organizational habitus is “a 
mechanism linking individual action and the macro- structural settings within which future action is taken. Habitus 
also links past fields to present fields through the individual actors who move from one to the next” (Emirbayer & 
Johnson, 2008, p.4). The “mechanism linking to individual action and the macro-structural settings” refers to 
internal technological tool (device) that links formative and summative assessment, past performance of an 
individual to future assessment, outcomes, or the past performance of a group to the norms of others groups, past 
and present. The “individual players” refer both to the instructors and to the students. In summary, the organization 
brings to it a habitus formed under specific past conditions, some of which will be shared with other members and 
some of which will differ from them substantially. 

As an institution of higher learning, institutional habitus is the band and integrity of the institution. Thomas 
(2002) emphasizes that “Institutional habitus should be understood as more than the culture of the educational 
institution; it refers to relational issues and priorities, which are deeply embedded, and sub-consciously informing 
practice” (p. 431). Further, it should be noted that institutional habitus is controlled by groups that control “symbols 
such as language, culture, and artefacts” (p. 430). This affects all strata of institutions from oversight panels to 
students. Institutional habitus most assuredly affects student retention performance and retention. Institutional 
habitus, while is most associated with branding, is similar to one’s personal habitus, but it conjoins the institution's 
dispositions, attitudes, and assumptions which are derived not only from the vision, mission, and core values and it 
goes beyond the simple structure of the organizational habitus. Institutional habitus speaks to the purpose of the 
institution, as underpinned by its culture and beliefs. Reay et al. (2001) defined institutional habitus as the impact 
of a cultural group or social class on an individual’ s behavior as it is mediated through an organization’(para. 

13). When the learner arrives at the academic online environment, he brings his personal habitus, but because he has 
chosen the institutional habitus as a personal match, and he is prepared to accept the habitus of the institution. 
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Personal habitus can be enriched and altered by the institutional habitus. “Quality matters” is the frame of the house; 
it is not the interior design. Instructional design is framed and relies on the pedagogy; the instructional designer 
takes the content and places it place it into the framework. However, while the institution maintains its institutional 
habitus, it recognizes that its brand is often times more powerful than the personal habitus. To many learners, 
higher educational environments must align with the individual’s perception of the academic and social match or the 
degree of academic and social integration from the institution (Tinto, 1975, 1993, 1997), and specifically to the 
institutional habitus. Tinto contended that a relationship must exist between the learner’s commitment goal to the 
institution and the institution’s academic and social characteristics and practices in order for the learner to obtain 
success. More importantly is the juxtaposition of personal habitus and institutional habitus within the online 
academic learning environment. If the online academic environment is not branded with the brands of the 
institutional habitus, the environment is blank and hollow and lacks the character of the institution. The 
environment then opens itself to hijacking by another individual’s personal habitus; the most vocal and the strongest 
habitus takes over. Brey (2007) contended that cybertechnology has certain built-in values and biases that are not 
always obvious or easy to detect. If this is the case, the institution must be forthright in not only branding the online 
environment, but also fulfilling the mission, vision, and core values of the institution through the institutional 
habitus. 

Bourdieu’s concept of capital is associated with both organizational habitus and institutional 
habitus. Cultural capital allows the learner to profit from formal and informal education in ways that those lacking 
in cultural capital cannot. As Bourdieu and Wacquant (1992) recognized that qualities of organizational habitus, or 
the academic and social accolades of an institution, offered the learner benefits, to which they stated, “one must 
identify the forms of specific capital that operate within it, and to construct the forms of specific capital one must 
know the specific logic of the field” (p. 108). Symbolic capital refers to positive recognition, esteem, or honor by 
relevant actors within the field. This applies to institutional habitus because symbolic capital consists of special 
authority that particular institutions are able to exert over the market, thus functioning as a form of credit, trust, or 
believe that those who have gained the institutional habitus bear more credence than others (Bourdieu, 2005). 

The connection that ties instructional design, feedback loop construction, and habitus so important hinges 
on this. Outcomes and assessments are linked to feedback loops that provide feedback for performance and 
instructional design (Emirbayer & Johnson, 2008). Habitus is basic to the growth of the individual learner, the 
instructor, and the institution. 

Feedback loops maintain the ebb and flow of the personal, organizational, and institutional habitus. 
Feedback loops force all parties who come saddled with personal and institutional values, objectives, and goals to 
analyze and evaluate their purpose in the symbiotic educational environment. When each loop supports the learner 
and the instructor, the loop creates a mutually beneficial support for desired learning. Students must adjust to using 
new media under different paradigms, and instructor must adapt pedagogies, methods, and attitudes to the new 
media, recognizing that the new media is a possible vehicle for teaching and learning. In the design and 
implementation of feedback loops within the context of virtual learning, it is essential that the instructional designer 
formulate them within the direction of the institutional habitus. In additional to these channels for learning, the 
designer uses a multitude of learning strategies and these courses are more than reading and responding to questions 
in the course. Feedback loops provide structured channels for iterative opportunities for students to move their 
learning closer to desired learning objectives or course outcomes. 
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Abstract 

Professors and K-12 teachers across all disciplines share the accountability of imparting information literacy skills, 
particularly those necessary for the research process, to their students. The English department certainly holds a 
large degree of responsibility regarding writing and information discovery, but the importance of these proficiencies 
is reinforced nicely when other departments require similar scholarly end products. In online classes, however, 
professors and teachers may hesitate to assign written papers because of the time required to grade them, the 
possibility of plagiarism, or the feeling that it is simply too difficult to explain the research process to students via a 
distance. Fortunately, digital tools can aid high school teachers, librarians, and college professors in this process. 

Introduction 

With the proliferation of internet resources, finding and evaluating sources for research has become 
simultaneously easier. . .and more difficult. The traditional paper-and-pencil method of writing a research paper may 
seem outdated, but its concepts are still critically important for students to master. Educators may be unaware of 
digital tools that help simplify the research paper process for students, and in turn, streamline their own workload. 
While not an exhaustive list, this paper will demonstrate some of the more beneficial features of EasyBib Pro, 
SafeAssign, and GradeAssist. 


EasyBib Pro 

EasyBib Pro allows students to create color-coded and keyword-tagged digital notecards in lieu of physical 
notecards. Students can then create a digital paper outline and drag and drop the notecards under the corresponding 
section ( Figure 1). Citation is easy; students simply copy and paste the URL of a website into EasyBib for auto- 
citation (Figure 2), but final verification is up to the student. When deciding if a source is credible, EasyBib ranks 
the 5,000 websites most frequently submitted for citation on factors such as bias, currency, completeness, and author 
credentials (Figure 3). The Analyze feature examines diversity of source types, database usage, number of sources 
for paper length, and website credibility to give an overall quality score to a student’s works cited page (Figure 4). 

With integration with Google Drive and Microsoft OneDrive, and Export function to Microsoft Word, 
along with the ability to generate a properly formatted MLA title page, EasyBib has multiple functions that make 
students’ lives easier. The program also has apps and extensions that integrate with Firefox and Chrome (Rapp, 
2011). Instructors like EasyBib too; by using the Share button, students add teachers directly to their projects, 
making live interaction possible. Groups can also co-edit the same bibliography. Most paid databases (including 
Gale and EBSCO products) have an “Export to EasyBib” button that allows the programs to work together. 

While EasyBib Pro is probably best suited for high school students, it could also have useful applicability 
for college freshmen, particularly those students in remedial or introductory classes. Undergraduate and graduate 
students who are experienced writers will still appreciate the assistance of EasyBib ’s free version in formulating 
MLA citations (the APA version costs money). EasyBib Pro can be purchased at www.easybib.com; institution and 
individual subscriptions are available. An alternate free tool that aids students with APA citation is 
www.citationmachine.net, which also scaffolds students on Chicago and Turabian styles. 
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Figure 1: EasyBib's Notebook and Outline Features 
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Figure 4: EasyBib assigning a bibliography score 
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SafeAssign 

SafeAssign is plagiarism detection software that is integrated within the learning management system 
Blackboard. Other separate software, such as Tumltln, performs similar functions. These programs generally charge 
a per-student subscription fee paid for by the institution. Professors can manually submit any assignment, such as a 
discussion board post, to check for originality. Projects such as research papers can be set up as SafeAssignments, in 
which the student bears the responsibility of submitting the paper for examination, not the teacher. 

SafeAssign maintains a database of papers submitted by students across all institutions that use its services. 
This alerts the professor if a student tries to use all or part of a paper already submitted by a third party. Similarly, if 
the student tries to self-plagiarize by using a previously written paper, SafeAssign will flag this attempt. SafeAssign 
also searches the internet and matches suspected passages against web pages ( Figure 5). 

SafeAssign has a Draft feature, in which students can preliminarily analyze their papers and see the 
potential plagiarism score. Since SafeAssign highlights any passage suspected of being unoriginal, the student can 
revise the paper before final submission to the professor. After all, some plagiarism is unintentional (Turnltln, 

2012) — students may not realize what they are doing is wrong — so showing them in detail which passages are 
unacceptable will help them to rework their papers and provide proper citation. While certainly penalties for 
plagiarism are appropriate, providing teachable moments and helping students avoid similar mistakes in the future 
will help them in the long run (Schweigert, 2013). One university found that plagiarism cases constituted 38% of its 
academic misconduct hearings (“Plagiarism Tops,” 2014), so the issue is not a minor one. 


289 





Figure 5: Sample SafeAssign report 
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GradeAssist 

GradeAssist is a Microsoft Word add-in that allows educators to apply detailed pre-populated comments 
specific to APA or MLA format directly in the body of students’ papers. For individual instructors, GradeAssist is 
$99 per license; bulk discounts are available for school- or university-wide purchases. Rather than typing the same 
comments over and over on students’ papers when they make similar mistakes, the instructor can simply select from 
a drop-down menu (Figure 6). The ability to create one’s own custom comments is another appealing feature of 
GradeAssist, but perhaps the best part of the program is that each comment references the page number of the APA 
or MLA manual on which the formatting error can be addressed in greater detail ( Figure 7). According to Volk, 
Pappas, Volk, and Warren (2011), “With less time spent grading the mechanics of dozens or hundreds of academic 
papers, educators are freed to focus on the substance of students’ work or to complete other professional tasks” (p. 
224). GradeAssist can be purchased at http://www.educo360.com; institution and individual subscriptions are 
available. 
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Figure 6: GradeAssist (Microsoft Word add-in) with sample comment 
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Figure 7: View in Word after GradeAssist comment is applied 


Abstract 

Professors and K-12 teachers across all disciplines share the accountability of imparting information 
literacy skills, particularly those necessary for the research process, to their students. The English department 
certainly holds a large degree of responsibility regarding writing and information discovery, but the importance of 
these proficiencies is reinforced nicely when other departments require similar scholarly end products. In online 
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Conclusion 

Thankfully, the Internet Age has provided professors, teachers, and librarians with digital tools that can 
help to restructure the research paper process. The process of discovery, critical thinking, and synthesizing of 
information has not changed, but the means by which students arrive at the final research paper product can be 
strengthened by using tools such as those discussed herein. As distance technology and Web 2.0 tools continue to 
evolve perhaps even less time can be spent on mechanics and administrative tasks and more time can be devoted to 
those processes that help increase student achievement. 
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Abstract 

There is a need for a holistic usability evaluation framework that accommodates social constructivist online 
courses. Social knowledge construction is at the core of such courses. These features may not be adequately 
evaluated using current frameworks. This qualitative research study examined the usability needs of a social 
constructivist online course. The intent was to identify elements of a usability evaluation framework specific to 
online social constructivist courses. Data was collected from an online course with 20 students and analyzed using 
phenomenography. The resulting analysis was compared to an existing framework for usability evaluations. One 
recommendation coming from the study was to classify usability concerns as either challenge or hindrance stress in 
order to eliminate hindrance stress and to appropriately select and pace challenge stress. Hindrance stress identified 
in this study arose from working across time zones, scheduling difficulties among collaborators, students dropping 
the course, and confusion of terminology. Challenge stress arose from interaction and collaboration with 
classmates, learning to navigate collaborative tools, and lack of examples of completed projects. Notably, the 
synchronous meetings were beneficial and effective at managing stress for some students. 

Introduction 

Usability evaluations are important for examining the flow and effectiveness of a course environment. 
Although traditional usability evaluations examine aspects like the ease of use and the look and feel of the course, it 
has been argued that instructional design features as well as the motivation to learn can, to a degree, be assessed 
using a usability evaluation framework (Zaharias, 2009). This paper examines the application of such a framework 
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to a social constructivist course and offers suggestions for courses that hinge on synchronous and asynchronous 
interpersonal interactions. 


Objectives 

Usability evaluation is an integral part of the course design process (Cennamo & Kalk, 2005; Piskurich, 
2006). However, the process of improving design through usability evaluation occurs more frequently in software 
and other consumer product development than in online course development (Fisher & Wright, 2010). Some 
usability evaluations for online courses may not address all aspects of the learners’ needs appropriately. When 
usability evaluation is applied to online courses, the learners’ pedagogical needs are not always addressed (Zaharias 
& Poylymenakou, 2009). 

The following research question guided the study: How can a usability evaluation framework designed for 
constructivist online courses be used to support the needs of social constructivist online courses? The framework 
that guided this qualitative study was based on Zaharias’s (2005) usability framework for online learning, which 
combined Web design, instructional design, and Keller's (1983) motivation to learn. 

Theoretical Framework 

Zaharias (2005, 2009) presented a usability evaluation for constructivist online courses with asynchronous 
interactions. This usability framework went beyond traditional usability evaluations to include not just usability, but 
constructs to measure the instructional design and motivation to learn. Within this framework, the parameters and 
attributes are as follows: 

• Usability: navigation, leamability, accessibility, consistency, and visual design 

• Instructional design: interactivity /engagement, content and resources, media use, learning strategies 
design, feedback, instructional assessment, and learner guidance and support 

• Motivation to learn: attention, relevance, confidence, and satisfaction. 

Zaharias referred to these as functional connections, cognitive (learning) connections, and affective 
(learning) connections (Zaharias, 2009). This framework goes beyond task completion to review application of key 
principles of pedagogy and learning theory (Zaharias & Poylymenakou, 2009). 

Zaharias (2009) offered the framework as a foundation from which to build. Alterations and adaptations 
were welcomed: “As Heller and Martin assert, this framework and the respective criteria can be ‘the floor not a 
ceiling for a series of guidelines that can be used to generate evaluation questions’ about an e-learning application” 
(Zaharias, 2009, p. 50). 

In a social constructivist course, as opposed to a constructivist course, it is recommended that the usability 
evaluation take into consideration the social context of the course (Blandin, 2003). “A convergence appears 
between cognitive approaches and sociological approaches which advocates the importance of cultural and 
sociological context for determining ‘usability’ of tools, in both its restricted and broad acceptances” (Blandin, 
2003, p.317). 

Social Constructivism 

Social constructivism refers to the Vygotskyian version of constructivism that includes collaboration with 
others as a key component. The zone of proximal development (ZPD) "is the distance between the actual 
developmental level as determined by independent problem solving and the level of potential development as 
determined through problem solving under adult guidance or in collaboration with more capable peers" (Vygotsky, 
1978, p. 87). At its core, the philosophy behind a social constructivist course is that knowledge is created when it is 
shared. “Many versions of [social constructivism] maintain that objects exist only after they enter communicative 
space” (Keaton & Bodie, 2011, p. 192). 

Additionally, in a social constructivist course, designers and instructors do not want to tell the students 
exactly what to do. Tam (2000) explained that the constructivist perspective “summons instructional designers to 
make a radical shift in their thinking and to develop rich learning environments that help to translate the philosophy 
of constructivism into actual practice” (p. 54). Students are often given a level of control over their own 
assignments while the instructor provides structure and scaffolding support (Tam, 2000). Assessment in a social 
constructivist course provides additional opportunities for student involvement. Students can work with their peers 
to evaluate one another’s work, which is expected to help the students foster a refined understanding of the content. 
This perspective means the designer and instructor relinquish some control of the course. 
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Usability and Stress 

While in an online class and using a learning management system (LMS), students deal with both 
computers and communications (Brown, Fuller, & Vician, 2004), which can be challenging and contain uncertainty 
on their own. Individuals may experience computer anxiety, communication apprehension or a combination of both. 
Brown et al., (2004) proposed that different types of computer applications may cause different types of anxiety. 

Anxiety has been shown to have a positive correlation with stress (Mughal, Walsh, & Wilding, 

1996). Anxiety is the concept or feeling; stress is a stimulus (Sarason, 1984). Stress can be defined as “a physical, 
chemical, or emotional factor that causes bodily or mental tension” (Stress, 1999, p. 1164). Low usability can be a 
factor in causing anxiety (stress) and impact on motivation, but not all stress is bad. Stress is considered helpful for 
individuals to learn (Joels, Pu, Wiegert, Oitzl, & Krugers, 2006). 

In learning, LePine, LePine, and Jackson (2004) divided stress into two categories: challenge stress and 
hindrance stress. Using terms from a work-related stress study, challenge stress includes “demands or circumstances 
that, although potentially stressful, have associated potential gains for individuals [while hindrance stress can] tend 
to constrain or interfere with an individual’s work achievement, and which do not tend to be associated with 
potential gains for the individual” (Cavanaugh, Boswell, Roehling, & Boudreau, 2000, p. 68). 

Examples of challenge stress included the number of projects, the level of difficulty and the amount of time 
needed to complete the work. Examples of hindrance stress included inability to understand class expectations, 
amount of time spent on “busy work”, perceptions that favoritism rather than performance affected final grades, and 
unnecessary obstacles encountered before completing a project (LePine et al., 2004). 

Mendoza and Novick (2005) conducted a longitudinal study of instructors as they learned to build websites 
and found their frustration level decreased as they progressed from novice learners to experienced users. The 
researchers recommended studying usability past the initial stages of a course startup. At the beginning of the study, 
most frustration was as a result of user error. As the course progressed, causes of frustration shifted to users having 
difficulty finding advanced features in the software application (Mendoza & Novick, 2005). The more they learned, 
the more they wanted to learn. A social constructivist course may lend itself to inherently having a certain level of 
frustration in users as they interact with one another and with the course to progress towards mastery and expert 
achievement. Course designers may need to identify usability items as related to either hindrance stress or challenge 
stress when evaluating such a course. Totally eliminating stress in a course is not desirable, as it would indicate the 
learners are no longer learning. 


Phenomenography 

Phenomenography is a qualitative method that is "more interested in the content of thinking than is 
traditional psychology" (Marton, 1986, p. 32). Meaning, the process of cognition is not as important as the meaning 
of the thoughts. Phenomenography deals with people's perceptions of the world, rather than explaining the world 
itself. It is not based on an objectivist epistemology, but rather on a phenomenological epistemology (Sandbergh, 
1997). "An effort is made to uncover all the understandings people have of specific phenomena and to sort them 
into conceptual categories" (Marton, 1986, p. 32). Briefly, the steps of the phenomenography method include: 

• Semi-structured interviews or equivalent data 

• Analysis of interview transcripts 

• Categorization of description based on the meaning of the text 

• Analysis of categories for hierarchical relationships (Marton & Booth, 1997). 

According to Marton and Booth (1997), "In principle, there is no impediment to using published documents 
as data, or even artifacts of other kinds that in some way serve as an expression of the ways in which people 
experience some part of their worlds" (p. 132). 

Phenomenographic analysis of student interviews was an important addition to content analysis by an 
expert (McCracken, 2002). Researchers showed that students in geology courses tended to have difficulty grasping 
the visualization and interpretation of three-dimensional maps (Edwards, 1986). McCracken (2002) examined how 
the learners in her study perceived this concept. The study found there was a disconnect between the instruction 
students needed in order to adequately comprehend the material and what experts thought the students needed; the 
lecturers involved in the study found students had not learned what instructors intended to teach them. 

Based on these results, McCracken modified course objectives and sequencing of instruction. Results in 
the subsequent course offering showed an increase from pre- and post-test scores of 30% (n= 22), with the four 
students who had the lowest pre-test scores showing the most gain. 
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Methods and Data Sources 


Study Context 

As stated above, this qualitative research study was designed to examine the usability evaluation needs of a 
social constructivist course. An online social constructivist course aimed to help teachers improve their basic online 
teaching and instructional design skills was selected as the context for the study. 

The rich environments for active learning (REAL) instructional design model (Grabinger, Dunlap, & 
Duffield, 1997) guided the course design and grounded the course activities in social constructivist principles. Some 
argue that the REAL model supports social constructivist online instructional design (Robinson, Phillips, Moore, & 
Sheffield, 2014). The REAL model encourages meaningful student learning through five key attributes: 

• Student responsibility and initiative 

• Generative learning activities 

• Authentic learning contexts 

• Authentic assessment strategies 

• Collaborative learning (Grabinger et al., 1997). 

Participants in a usability study should be real users performing real tasks (Dumas & Reddish, 1999; 

Genov, 2009; Rubin, 1994). "The closer that the scenarios represent reality, the more reliable the test results" 

(Rubin, 1994, p. 179). The course was designed around two major projects in which the students worked in teams to 
design online learning units that each team would deliver to their peers. The authentic and collaborative nature of 
these assignments highlights the social constructivist aspects of the course. The course content, activities, and 
interactions were based largely in the Moodle LMS. Weekly synchronous meetings were also held through the web 
conferencing software Adobe Connect. 

Rubin (1994) encouraged the use of rewards to motivate users. Motivation was a concern of the course 
designers because this initial offering of the course was free for volunteer participants. Attrition rates in such 
courses are especially high (Zaharias & Poylymenakou, 2009). Digital badges were awarded to students after the 
completion of certain tasks or challenges in an attempt to maintain or stimulate student motivation. These badges 
included metadata describing the evidence of the achievement, the skills or knowledge the learner needed in order to 
earn the badge, and the organization offering the badge. The badges functioned as a certification of achievement of 
certain benchmarks reached in the course. 

Participants 

Research indicated that usability testing works best when performed on four or five users (Nielson & 
Landauer, 1993; Virzi, 1992), for both cost and results. However, the course design called for six student teams. In 
order to evaluate collaboration, community building, and social knowledge construction in the course, as well as to 
offer a cushion against attrition, 20 class members were recruited for the online course in this study. 

Faulkner (2003) offered support for larger groups of participants in qualitative usability evaluations. She 
challenged the use of modeling to arrive at the recommended number of four or five users. She demonstrated with 
users instead of models the inadequacy of that number, showing that ten to twenty users found a higher percentage 
of usability problems. 

The course was offered for free and without formal credits. Participants were recruited through Moodle 
forums, conferences, and the researchers’ social networks. All participants joined the course voluntarily. 

Participants were aware that they were joining a newly created course and that their participation in the course’s first 
offering would help to refine the course for future use. Participants also knew that the course and their participation 
in the course would be studied. 

Participants in this study were adult professionals in the field of education. Some were teachers, some 
consultants, others worked for educational firms. Participants ranged in experience with online learning. Some were 
highly experienced and had taken and taught many online courses in the past. Whereas, others were taking their first 
online course. 

Data Sources 

Sources of data from the course included weekly feedback from the participants, discussion forums, and 
activities (e.g., quizzes, surveys, and assignments). These data offered a glimpse into the natural workings of the 
online learning environment as well as direct comments and questions from the users about the usability of the 
environment. 
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Data Analysis 

A phenomenographic approach was selected to analyze the data. The categories of description and 
hierarchical relationships resulting from the phenomenographic analysis were compared and contrasted with the 
Zaharias (2009) framework for usability evaluation. First, the data from the course was exported and compiled into 
a spreadsheet in preparation for coding. Utilizing a computer program to randomly determine pseudonyms, all 
names of students in the course were replaced with pseudonyms to anonymize the data. The data were divided into 
thirds and coded individually by three researchers. The Zaharias framework containing 16 usability parameters was 
utilized as the basis for this first round of analysis. Each student question or comment was analyzed for placement 
into one of the 16 parameters. If stress was identified, an additional classification was applied: hindrance stress or 
challenge stress. These classifications comprised this first round of coding. 

A second round of coding was undertaken in order to establish inter-rater agreement. The three researchers 
exchanged coded sections of the data. The initial codes and classifications were reviewed a second time. When 
disagreements arose, the researcher made comments on the codes and instances in questioned and suggested changes 
to the coding classifications as needed. Several meetings occurred during this time in which the researchers 
discussed various placements of categories and subcategories. This process allowed for consistency in categorical 
placements. The three sections of coded data were then combined for analysis. 

Results 

The parameters within the cognitive (learning) connections category contained the majority of instances 
overall, specifically within the sub parameters of interactivity, content and resources, and learning strategies design. 
The references and instances of both hindrance and challenge stress were highest within the learning strategies 
design parameter. This parameter offers a way to measure fundamental principles of learning theories and 
pedagogies (Zaharias, 2009). Among other things, the learning strategies design parameter allows for measurement 
of peer-to-peer interactions. Two of the descriptors under this parameter emphasizing peer-to-peer interactions are: 
1) “The courses provide opportunities and support for learning through interaction with others (discussion or other 
collaborative activities)”, and 2) “The courses include activities that are both individual-based and group-based” 
(Zaharias, 2009, p. 52). 

Likely due to the social constructivist nature of the course, the peer-to-peer interactions within the learning 
strategies design parameter featured prominently in the data set. The course offered opportunities throughout the 16 
weeks for learning through interaction with peers and the instructor with the use of discussion threads and individual 
activities (asynchronous) and weekly online meetings in Adobe Connect (synchronous). Various authentic 
collaborative learning activities were developed for the students, including one extensive group project. Students 
expressed concern with: 

• Working across numerous time zones due to the global nature of the group (hindrance stress), 

• Scheduling difficulties (hindrance stress), 

• Team members dropping out (hindrance stress), 

• Inefficiencies of working collaboratively (challenge stress), and 

• Lack of concrete examples of projects (challenge stress). 

In this course, students were allowed to self-select their groups. Instructors did not control for the time 
zone differences and allowed participants from around the world to enroll. The challenges that this brought to the 
group activity may have been a hindrance for some students to complete the course. 

Some of the student statements referencing interpersonal interactions could have been classified under 
more than one framework classification. For example, one student described anxiety about working in a group. This 
statement could have been placed under learning strategies design (cognitive connection) or satisfaction (affective 
connection). Similarly, one student requested an earlier deadline for choosing collaborative partners. The 
interpersonal interaction aspect of this comment aligned with learning strategies design (cognitive connection) while 
the procedural side of this comment aligned well with content and resources (cognitive connection): “Content is 
organized in an appropriate sequence and in small modules for flexible learning” (Zaharias, 2009, p. 52). In another 
example, a student suggested the enabling of emoticons in the discussion forums to assist in conveying tone and 
intention behind comments. The interpersonal interactivity aspect of this suggestion fit in learning strategies design 
(cognitive connection) and, at the same time, the technical recommendation fell under visual design (functional 
connection). 

The parameter with the second highest coding instances was content and resources. This parameter 
measures the quality of written and otherwise represented ideas in the course using “criteria such as credibility. 
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accuracy, objectivity, coverage and currency” (Zaharias, 2009, p. 47). In this course, content was organized in 
weekly modules and resources were contained within online books. Learning objectives were provided for the 
students at the beginning of each online book. Links to readings and assignment instructions were also included in 
the books. 

One area of importance noted by students within the content and resources category was the terminology 
used by the course designers. Confusion of terms was considered by the researchers as a hindrance stress. There 
was confusion over the terms assessment, evaluation, peer review, and rubric. Students from different parts of the 
world had different understandings of these terms. The instructors and students worked together within the 
discussion forums and synchronous meetings to clarify the terminology and to come to a common understanding of 
definitions. 

Another prominent theme in the data was the challenge stress of learning to effectively use new technology 
tools incorporated into the course. This challenge stress fell under several framework parameters including learning 
strategies design, navigation, and visual design. The technology tools discussed in the data were the breakout 
activity in a synchronous meeting using Adobe Connect and the discussion forums, database activity, and workshop 
activity in the Moodle LMS. 

Positive feedback 

The amount of positive feedback noted by students throughout the 16 weeks emerged in the analysis of 
data. In particular, the students expressed positive feedback for the synchronous sessions and found value in these 
meetings for tying the activities together and clarifying expectations. One student explained that these meetings 
were grounding and helped the student remain accountable and connected with the teacher and peers. It was also 
noted that the synchronous meetings were excellent for collaborating and allowed students to talk through things in 
voice rather than text. A smaller number of comments described the synchronous meetings as a waste of time. 
Presumably, those without questions benefited less than those in need of clarification. Additionally, students were 
pleased that the instructors emphasized pedagogy over technology. Students also appreciated the amount, level of, 
and promptness of feedback. 


Discussion and Conclusions 

Usability evaluations may benefit from the identification of hindrance and challenge stress (see Figure 1). 
Usability concerns that result in hindrance stress should be mitigated while those resulting in challenge stress should 
be appropriately supported and paced. 


Hindrance Stress/Challenge Stress 



Usability Evaluation Framework (Zaharias, 2009, p. 50) 
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Figure 1. Classification scheme for analysis of student experiences. 
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Hindrance stress detracts from the learning experience. It prohibits the learner from progressing efficiently. 
The concept of hindrance stress is well aligned with traditional usability evaluations. Hindrance stressors build 
barriers to achievement of the goals and objectives of a course. These stressors should be identified and reduced 
whenever possible. As this study indicated, hindrance stressors might include working across different time zones, 
scheduling difficulties among collaborators, students dropping the course, and confusion of terminology. Course 
designers should be sensitive to students’ geographical locations and make accommodations so that students can 
work without excessive stress from geographical barriers. Course designers should also be aware of differences in 
language usage. A course glossary might be included to alleviate conflicting interpretations of terms and to raise 
awareness of the different uses of terms. Course designers should make a plan for adjusting or reallocating 
collaborative work in the event that students quit the course prematurely. 

Further, course designers should not underestimate the energy and time required to build relationships 
among classmates to the degree that students can satisfactorily and effectively collaborate. For some students, 
limiting the number of people in a collaborative group may be beneficial. They may benefit from working in an 
online synchronous meeting setting where voice and video options can enhance the collaboration. Other students 
may prefer and benefit most from working independently. Likewise, Zaharias and Poulymenakou (2007) noted that 
“catering for cultural diversity seems imperative in the design of e-learning courses or technologies for international 
use” (p. 748). The approach to grouping students is something to consider for course designers and instructors. 

At the same time, usability evaluation should examine and evaluate the intensity of challenge stress. 

Pacing challenges could be thought of as scaffolding: giving learners manageable challenges that increase in 
difficulty. As the challenges become more difficult, the learners gain more understanding, skills, and the confidence 
that allows them to tackle larger challenges with a tolerable stress level. The challenge stress experienced by the 
learner should remain more or less constant as the challenges increase. Carefully paced challenge stresses may 
balance feelings of frustration with motivation to overcome challenges. 

The concept of challenge stress is perhaps better associated with learning theories and pedagogy than 
traditional usability evaluation. Zaharias (2009), however, argues that the assessment of pedagogy implementation 
should be incorporated in a usability evaluation framework. The implications of this study indicate that instructors 
should consider limiting and pacing the challenge stressors they introduce to the class. Students who are new to 
online learning may experience excessive stress due to the challenge of learning in a new environment. Many tools 
exist to support social constructivist collaboration; however, students may feel overwhelmed by the technology and 
learning how to effectively use a large number of different tools. These students may not benefit from a myriad of 
tools for collaboration (e.g., wikis, databases, discussion forums). Instead, frustration and dissatisfaction may result 
from such a high learning curve. Adequately training students to use the tools is expected to improve students’ 
success and satisfaction in an online course. Thus, the best teaching tools for the job are only the best if their use 
does not overly stress students. Reducing the variety of tools along with heavily supporting the collaborative 
learning activities in a social constructivist course may improve the pacing of challenge stressors. The results of this 
study suggest limiting and pacing the amount of social interaction and collaboration tools in an online course. 

Additionally, the findings of this study suggest that synchronous meetings may be beneficial in supporting 
students as they are introduced to assignments and as they collaborate. Synchronous meetings allow students to 
voice their thoughts and concerns and receive immediate feedback. Additionally, instructors and more experienced 
peers are on hand to help students having technical difficulties. Incorporating synchronous meetings may help 
instructors manage stress experienced by students. 

Further research towards a framework of usability attributes to evaluate a social constructivist online course 
may help instructors meet evaluation needs in ways that better support the nature of social constructivist courses. 
Examining usability through the lens of challenge and hindrance stress sheds light on the type and import of 
problems and struggles that students experience in an online course. Identifying hindrance stress in order to 
eliminate barriers to learning fulfills the traditional role of usability evaluation. Identifying challenge stress in order 
to understand the foundation and nature of the anxiety experienced by students, so the pace and context of 
information can be most effective and efficient under the circumstances, helps to bring learning theories and 
pedagogy into the realm of usability evaluation. 
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Games, Simulations, and Pedagogy 

Today’s P-12 students are immersed in a world of games and simulations, Internet surfing, social media 
exchanges, and texting. Rarely does one look around and not notice individuals, even those as young as four or five, 
on some type of technological device. Some companies even target younger children by marketing his or her own 
first laptop/tablet. Consequently, with these increasingly ubiquitous technologies, along with the increasing 
requirements for assessment in education, serious consideration should be given to the use of games and simulations 
as assessment tools. 

As P-12 education begins to integrate technology, such as games and simulations into the classroom 
experience, it is essential that the use of these delivery methods indicate student progress and improvement in the 
classroom (Bellotti et ah, 2013). Traditionally, assessments have often been relegated to tests, quizzes, and faculty 
observation of individual presentations and group assignments; however, games and simulations supply these same 
assessment elements through player development within the game or simulation using scores, resources collected, 
levels achieved, and progress during the session (Bellotti et ah, 2013; Gredler, 1996; Moreno-Ger et ah, 2008). 
Games and simulations provide the opportunity for creative and critical thinking, problem solving, collaboration, 
and resource management as a player advances through multiple levels while persevering through obstacles. 

Previously, the pedagogy of P-12 education has been generally teacher-centered with lectures, 
presentations and textbook readings; however, more recently, students want more involvement and ownership of 
their own education with more technology integration (Shute & Ke, 2012). This shift in student expectations 
supports the integration of games and simulations to provide a more immersive education. For instance, games and 
simulations promote transformative learning as students are interactive in the environment, motivated to progress 
through levels, and build skills such as teamwork, collaboration, decision making, problem-solving, communication, 
negotiation, resource and time management, and promote higher order thinking through modified challenges 
(McClarty et al., 2012; Shute & Ke, 2012). Games and simulations are based on rules of play and progression that 
provides self and/or peer assessment just as teacher-centered instruction is based on lesson plans outlining goals and 
objectives with a quantitative exam or qualitative instructor review. Flowever, games and simulations provide instant 
feedback that allows the player to shift strategy and performance in order to progress; whereas, exam scores and 
instructor comments are typically delayed and can be ineffectual when received because the student has moved on to 
the next subject. 


Why Games and Simulations are Well-Suited as Assessments 

Games and simulations as assessments have already had success in fields such as the military and medical 
fields (Gredler, 1996). For example, commercial and recreational pilots have learned basic flight skills, decision 
making, and problem solving skills from Microsoft® Flight Simulator™ (Mislevy, 2011). This simulator furnishes a 
realistic environment of surrounding terrain, weather conditions, altitudes, fuel levels, and flight patterns while 
providing instant feedback that enables the individual to make any necessary adjustments. This data is viewable and 
can be used to assess how the player progressed and whether the flight landed safely or crashed (i.e., passed or 
failed). 
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In a P-12 setting, due to the hands-on nature of games and simulations, students are stimulated and engaged 
in the learning process; therefore, the student has a greater tendency to recall information from games and 
simulations more so than a lecture or presentation (Klassen & Willoughby, 2003). Games and simulations as 
assessment also provide what Shute (2011) refers to as “stealth assessment” where players immerse themselves into 
the environment, focusing on advancing levels and collecting resources, which can be considered an active 
assessment since actions of problem-solving and decision making are included rather than simply the regurgitation 
of information. In addition, when students do not know they are being assessed, like when playing games, they tend 
to score higher since the pressure and fear of testing is not considered a factor for them, especially in an entertaining 
environment like a game (Shute & Ke, 2012). 

Games and simulations for assessment are successful when measurable data can be collected on what is to 
be taught or demonstrated, what knowledge or skill is expected to be obtained, and what specific measure or 
evidence from the learner indicates acquisition (Mislevy, 2011). Shute and Ke (2012) suggest that games and 
simulations for assessment, based on an evidence-centered design model that outlines student learning outcomes and 
objectives that support levels of achievement, can be valid and reliable. Furthermore, Mislevy (2011) suggests a 
five-layer framework for evidence-centered design of assessment that promotes games and simulations built on 
domain foundations and expectations of student performance. Bellotti et ah, (2013) state that games and simulations 
provide measurable assessment data based on game play and eliminates the traditional method of teaching to the test 
as games and simulations provide branching experiences for learning. Del Blanco, Torrente, Marchiori, Martinez- 
Ortiz, Moreno-Ger, & Fernandez-Manjon (2010) add that games and simulations integrated in learning management 
systems also provide tracking of student progress while providing performance reports that provide quantitative data 
of learner interaction and progress. Fu, Zapata, and Mavronikolas (2014) further substantiates games and 
simulations provide not only data of the final response or outcome but can provide detailed data of the processes 
generated throughout which provide feedback for specific areas for improvement. 
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Abstract 

This case study investigated the process which 119 faculty members went through as they transitioned from using 
Desire to Learn (D2L) Learning Management System (LMS), to using Canvas LMS. Other than analyzing issues 
faculty members encountered while navigating various aspects of the Canvas interface, the study also analyzed 
technological issues faculty members come across while integrating software applications that work in Canvas. 
These applications included: Panopto, Voice Thread, Respondus Lockdown Browser and Turnitin. The study 
presents implications to faculty members, instructional designers and administrators. 

Keywords: Learning Management System (LMS), faculty development, instructional design, online learning, case 
study. 


1. Introduction 

Internet based Learning Management Systems (LMSs) such as Blackboard, Moodle, WebCT, Canvas, 
Scholar, and Desire2Learn are some of the popular Internet technologies that have been supporting distance, face-to- 
face and hybrid/blended teaching-learning processes. (Dahlstrom, Brooks, & Bichsel, 2014; McGill & Hobbs, 2008; 
Connolly, MacArthur, Stansfield, & McLellan, 2007; El Mansour & Mupinga 2007; DeNeui & Dodge 2006). A 
LMS can be defined as “a self-contained webpage with embedded instructional tools that permit faculty to organize 
academic content and engage students in their learning" (Gautreau, 2011, p.2). Again, Alias and Zainuddin (2005) 
defined a learning management system (LMS) as “a software application or Web-based technology used to plan, 
implement, and assess a specific learning process” (p. 28). Another definition still, looks at LMSs as web-based 
technologies that provide instructors with a way to create and deliver content, to monitor student participation and 
engagement, and to assess student performance online (Venter, van Rensburg, & Davis, 2012). (Venter, P., van 
Rensburg, M. J., & Davis, A. (2012). What is common in all these definitions is that an LMS is a web-based 
application that supports teaching and learning by enabling instructors to create and organize content for learners. 

LMSs are a technology that enables the communication of course expectations through various resources 
such as a syllabus, as well as of assignment instructions, grades, and instructional materials (Rubin et al., 2010). As 
Bonk and Reynolds (1997) observed, the paradigm shift from traditional educational environments to online 
educational environments in higher education can also be seen as a challenge to create an active and interactive 
learning environment, one which gives the learner opportunity to engage and think in multiple ways. In a study that 
investigated technology adoption into teaching and learning by university faculty for example, Nicolle (2005) found 
the link between effective teaching and the use of technology to be critical in helping faculty through the process of 
integration. As Baia (2009) observed, university faculty members are concerned with effective teaching, hence if 
they perceive technology as having a positive impact towards this effort, they are likely to get motivated to integrate 
it in their teaching. 

Several scholars have investigated how faculty and students value and use a LMS in teaching and learning. 
Pajo and Wallace (2001) stressed that successful integration of technology in teaching depends not only on 
availability of technology but also on how instructors embrace and use it. In a survey on faculty attitudes on 
technology, most faculty reported using a LMS, but using limited features as follows: Posting course syllabus 
(78%), recording grades (58%), communicate with students (52%). Only 20% of faculty reported using the LMS to 
record lecture content (Jaschik & Lederman, 2014). Recent LMS studies suggest that a variety of system issues like: 
Suitability of design in screen and system, easiness of course procedure, interoperability of system, easiness of 
instruction management and appropriateness of multimedia use, flexibility of interaction and test, learner control, 
variety of communication and test types and user accessibility as important LMS features that directly or indirectly 
benefit LMS or e-learning users and influence their attitudes towards LMS (Fathema & Sutton, 2013; Kim & Leet, 
2008; Weaver et. al,2008; Panda & Mishra, 2007; Pituch & Lee, 2006; Russell, et.al.,2003). The literature further 
indicates that other studies on LMS have focused on how faculty and students value and use an LMS in teaching and 
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Learning. Yet more studies have focused on faculty perceptions on the whole transition process to a new LMS. 
Against this background, it was found necessary to also identify specific technological issues faculty encounter 
while transitioning to a new LMS and how best such issues can be mitigated. 

Purpose of Study 

LMSs enable the communication of course expectations through various resources such as a syllabus, as 
well as of assignment instructions, grades, and instructional materials (Rubin et ah, 2010). The present study 
investigated the process which 119 faculty members at a state university on the southern part of United States went 
through as they transitioned from using Desire to Learn (D2L) Learning Management System to using Canvas. The 
study did not only analyze issues faculty encountered while navigating aspects of the Canvas interface per-se, but 
also issues they came across while integrating other technological applications that work in Canvas. So, apart from 
general Canvas interface challenges, intricate issues emanated from using four applications leading to the 
subsequent revision of workshops for the future. The study presents implications to instructional designers, 
administrators and faculty members on the intricate process of implementing new educational technologies and the 
best way to manage learning management system transition all together. Specifically, the study sought to answer the 
following questions: 

• What general Canvas interface issues did faculty members face in transition from D2L? 

• What issues did faculty members encounter while integrating various software applications in Canvas? 

• What implications did these issues have on preparing future faculty development workshops? 

1. Research Design and Methodology 

1.1. Context and Participants 

A State University on the southern part of United States had been using Desire to Learn LMS in the past. An 
administrative decision led to a switch. While the University administration made the decision to switch, campus-wide 
consultations with faculty members were made for the selection of a new LMS from a short-list. Canvas was in the 
process selected to be the new LMS the University would be adopting. The Center for Excellence in Teaching and 
Learning was charged with the task of training faculty members to using Canvas. Between August 2013, and January 
2014, a total of 1 19 faculty members attended four different workshops that primarily focused on training them on how 
to use and navigate various course related components in Canvas. A total of four workshops covering different aspects 
of Canvas LMS were designed and taught repetitively for a week in October, 2013. Workshops were taught again in 
January 2014, again, repetitively for a week. The majority of the faculty who took training were those who taught 
online. However, some faculty members who never taught online also attended the workshops. And so from a total of 
119 faculty members, about 95% of these taught online and only about 5% did not. 

The four workshops were divided based on sections in the Canvas interface. Again, inevitably, faculty 
members had to integrate other software applications that work in an LMS. Due to limited time allocated to running the 
workshops, these applications were not covered deeply in the four workshops since priority was placed on training 
faculty members on using features of the new Learning Management System rather than add-ons. To that end, it would 
be found that mastering the latter would pose more challenges to faculty members than general learning of the Canvas 
interface. A faculty member who is also the Teaching and Learning Coordinator prepared the workshops and ran them 
repeatedly in collaboration with the Director of Excellence in Teaching and Learning. As expected, after scheduled 
workshops were done, faculty members encountered contextual issues while using Canvas. Faculty members would 
call the Teaching and Learning Coordinator for assistance with various issues while those who needed more specialized 
assistance would make one-on-one appointments. This was found to be a helpful strategy since it was naturally found 
that there were varying degrees of technology proficiency among users. 

1.2. Data Collection 

Data for this qualitative study was collected by compiling case study reports on a day-to-day basis. A case 
study is an empirical inquiry that investigates a contemporary phenomenon in depth and within its real-life context, 
especially when the boundaries between phenomenon and context are not clearly evident, ft is also concerned with 
studying the phenomenon in context, so that the findings generate insight into how the phenomenon actually occurs 
within a given situation (Creswell, 2009; Yin 2009). A google document (google doc) was created and all issues faculty 
raised during workshops, on the telephone and in one -one- meetings were systematically recorded on it. After formal 
workshops, faculty continued to make calls and make one-on-one meetings with the Teaching and Learning 
Coordinator. During these calls and one-on-one meetings, faculty members would raise different issues with Canvas 
that were addressed by the Teaching and Learning Coordinator. These issues were all recorded on the google doc. The 
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recorded notes focused on technological issues users encountered while learning how to navigate the general Canvas 
interface. The issues ranged from general Canvas interface queries to more complex issues having to do with 
integrating four software applications namely; Panopto, Voice Thread, Respondus Lockdown Browser and Turnitin. 
While some of these applications such as Respondus lockdown browser had been used by faculty members in Desire to 
Learn (D2L) LMS, they still posed a variety of integration issues in Canvas. Notes relating to issues faculty members 
encountered while navigating the general Canvas interface and while integrating the four external applications were 
later complied into a detailed case study report from which this study was developed. 

2.2 Researcher Stance 

The Teaching and Learning Coordinator, a faculty member who performs instructional designer duties was 
the researcher in this study. The researcher, therefore, had the insider’s perspective. Initial role began with preparing 
workshops by way of synthesizing all content to be taught in the workshops. The researcher went on to facilitate the 
workshops by collaborating with the Director of Center of Teaching and Learning. All questions during workshops 
were addressed by the researcher (The Teaching and Learning Coordinator). The researcher further recorded all the 
issues raised during workshops on the google document that was created for data collection purposes. After formal 
workshops, users either made calls to the Teaching and Learning Coordinator (the researcher) or made one-on-one 
appointments. Issues that users inquired about during these calls and meetings were recorded on the google doc by the 
researcher. The researcher analyzed the data from which the present study was created. 

1.3. Data Analysis 

Data collected from arising issues was analyzed based on two categories. Category One comprised of general 
Canvas interface issues while Category Two comprised of issues deriving from integration of four software 
applications namely; Panopto, Voicethread, Respondus Lockdown Browser and Turnitin. While there was an enormous 
amount of data collected over months of Canvas delving, whole text analysis was used to examine the notes recorded 
on the google doc. This technique requires the researcher to fully understand the purpose of his or her study to enable 
them study the data continuously in order to identify specific codes. This procedure for analysis was developed by 
Glaser and Strauss (1967) and Strauss and Corbin (1998). From the theme: “Technological issues arising from faculty 
use of a new learning management system,” the two categories are presented in tables 1 and 2 below: 

Table 1: General Canvas Interface Issues Encountered by Faculty Members 

• How do I edit and change course dates? 

• What do I do to give true extra credit in Canvas? 

• How to enter paper submission grades into grade book. 

• How do I enable students attach files to discussion posts? 

• What do the various quiz icons stand for? 

• How do I import question banks from an external source. 

• Moderating a quiz to give more time to students. 

• Is it possible to reinstate an exam I deleted accidentally? 

• Is it possible to reinstate grades for a student who was removed from my course? 

• How is a new column created in the grade book? 

• How do I weight my final grade based on various graded events? 

• Holding quiz results from student view. 

• How do I generate an attendance report in Canvas? 

• How do I save and print speed grader comments along with the submitted paper? 

• How do I add a new set of student groups in Canvas? 

• Viewing course analytics without going to the “people” page. 

• How is an external calendar feed added to a Canvas account? 

• How do I use fudge points in speed grader? 

• How do I give a letter grade? 

• I cannot see course modules in student view even though I have enabled them for students. 

• My announcements are not going out but I feel like I have done everything correctly. 

• I have old assignments from two years past still appearing under syllabus and was confusing students. 

• Is it possible to be gradually giving feedback to the same assignment throughout a semester? 
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Table 2: Issues Arising from Integrating External Applications into Canvas 

Voice Thread 

• Students are not able to see one another’s Voice Thread projects, yet all settings were done correctly. 

• Students able to create their own voice threads, they can see the instructor’s, but cannot comment on one 
another's. 

Voice Thread project not available on campus computers, yet students are able to view them from 
outside. 

• In going to student view, when I click on the Voice Thread project in one of my modules, I have to sign 
in. Normally, I am not supposed to do that. 

• Unlike with audio comments, I cannot upload my video comment to class Voice Thread project. 

Panopto 

• Recorded a project in Panopto but do not know where to go in order to save it. 

• Students cannot create Panopto projects to make recordings of their work. 

• Panopto asks students to sign in while in the course. 

• Cannot see my Panopto video when in Internet Explorer. 

• How best do I use Panopto recordings to run a flipped classroom? 

Turnitin and Grademark 

• When I use Grademark to give feedback, students cannot see comments I make. 

• How do I enable students print out Grademark grades and my feedback? 

• How do I have Turnitin process an assignment that was submitted before I enabled the app in my 
course? 

• I can see my Grademark comments but students cannot. 

• Submissions not generating an originality score with Turnitin due to a “class does not exist” error. 
Respondus Lockdown Browser 

• How to align quiz content between Respondus and Canvas? 

• I have disabled the Lockddown browser link in my course so students do not have access to it but even 
myself cannot locate it. 

• Student can't get exams in Respondus to open fully. 

• How do I print Canvas tests using Respondus? 

• Receiving error message "unable to connect to the testbank network server." 

• Several students email me from my class last night stating that Respondus wouldn't work. Something 
about saying it had no internet connection but they could get to the internet just fine if they were not 
using that. Was there a server issue? 

• I have had multiple students stating that Respondus didn't prompt them to use the webcam. Settings for 

Respondus Monitor look correct. Why is it allowing them to take the quiz without the webcam? 


Table 3: General Observations 

• General Canvas interface questions came up during workshops, more contextual, specific issues arose 
while using Canvas after workshops. 

• Six months after scheduled workshops, basic issues to do with importing content from the expiring D2L 
still came up. 

• Most serious issues arose from faculty use of external applications in Canvas. 

• Most users who had reservations about Canvas at the beginning, ended up liking it later when they 
started using it. 

• There was variation in speed of mastery of the new LMS among users. 

• In due course, more faculty who did not teach online ended up wanting to learn Canvas. 

• Faculty members who did not teach online found features like the gradebook very helpful in managing 
student grades. 

• Issues faculty members encountered helped designers to go back and formatively evaluate and revise 

workshops. 
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2. Findings 

2.1. Research Question 1: What general Canvas interface issues did faculty members face in 
transition from D2L? 

Data mostly collected from faculty calls and one-on-one meetings with the Teaching and Learning 
Coordinator (refer to Table 1) indicated that as faculty members went to apply the skills they had learned in the 
workshops, contextual issues arose as they navigated various aspects of the Canvas interface. An instructional 
design role is about problem solving, it can be argued. Some of the Canvas issues faculty members encountered 
were resolved within minutes simply by showing an instructor how it is done. These were categorized as Level One 
issues. Other issues, however, took a reasonable amount of effort to be resolved. These were categorized as Level 
Two issues and were mostly resolved face-to-face with a faculty member. Yet other issues took a substantial amount 
of research including involving contacting Canvas on behalf of faculty members or with them. The latter were 
categorized as Level Three issues. Level One issues like how to change course dates or meaning of the various quiz 
icons would be resolved with a simple explanation or demonstration. Level Two issues like how to give true extra 
credit or how to import question banks form an external source or indeed how to moderate a quiz in order to allocate 
more time to specific students required a well prepared tutorial to faculty members. Depending on how quickly a 
faculty member would master the action, steps would be repeated mostly in a one-on-one meeting until the user was 
able to perform the required action on their own. Even more interesting were Level Three issues such as importing 
quiz banks from an external source, reinstating a deleted quiz, reinstating grades for a student who had been dropped 
from a course, and others. While the first task, importing quiz banks from an external source would be done with 
the faculty member, the other two issues, reinstating grades and deleted quizzes could not be handled at the level of 
Canvas access of a college administrator. These would therefore require the Teaching and Learning Coordinator to 
contact Canvas by either creating a ticket or using the chat feature. Later on, faculty members were walked through 
creating tickets of their own and being able to use the chat feature to have such type of Level Three issues resolved. 
This strategy allied well the philosophy of doing it together in instructional design (Dick, Carey, & Carey, 2009). 

Figure 1: The Three Levels 
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2.2. Research Question 2: What issues did faculty members encounter while integrating various 
software applications in Canvas? 

All issues that arose from integrating the four external applications into Canvas were categorized as Level 
Three issues due to the depth of their complexity. From a general perspective, the integration of these applications 
posed more challenging issues to faculty members than general Canvas interface use. 

2.2.1. Voice Thread 

A Voice Thread is a dynamic, living conversation space that can be altered anytime. Basically, the 
application is about creating collaborative space with video, voice and text commenting (Voice Thread, 2015). 

Issues with Voice Thread like students not being able to see one another’s projects when the instructor felt 
they had done all the settings correctly posed an intricate challenge to solve. The same applied to issues like a Voice 
Thread project not available on on-campus computers. These issues required very systemic diagnosis which began 
with asking the instructor for very specific details of what exactly was happening. The diagnosis would take place 
through email or telephone. Once details of the problem were made clear, the Teaching and Learning Coordinator 
would identify solutions that would essentially solve the problem. Again, the whole process would be done while 
doing it together with the faculty member rather than doing it on their behalf. Interestingly though, a few faculty 
members would ask that the problem be solved for them rather than work through it together. This simply 
highlighted varying faculty preferences when it comes to solving technological issues. Procedurally, issues that 
could not be solved at the level of access of on-campus Voice Thread administrators would require creating a ticket 
with Voice Thread engineers who would in turn examine the problem and offer solutions. 

2.2.2. Panopto 

Panopto is a software for businesses and universities that makes it easy for anyone to record, live stream, 
and share video (Panopto, 2015). In Education institutions, Panopto is mostly used for lecture capture. Faculty 
members would normally record a lecture with video and/or PowerPoint and in turn post it in a module for students 
to view at any time. For example, an instructor wanted to find out how best they could use Panopto to run a face-to- 
face flipped classroom. This scenario actually justified the application’s valuable use. While working with the 
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instructor, the Teaching and Learning Coordinator suggested that the best way to do it would be to make Panopto 
recordings of all lessons and post them in a module, for example, and set each lesson recording to open a few days 
prior to they would be taught. That way, students would watch the stream prior and in turn discuss it when they 
come to class. Another interesting issue involved students not being able to make Panopto recordings themselves as 
the instructor had asked them to. While running a diagnosis of this issue, it was found that the instructor had actually 
made the settings in such a way that only teachers could create projects and not students. This made sense in the 
context considering that the University had purchased licensing for the application primarily for lecture capture and 
for student view. Finally, some issues that faculty members encountered with Panopto made the Teaching and 
Learning Coordinator offer the advice that sometimes it was a browser issue as the application did not normally 
work well in Internet Explorer. To that end, Google Chrome and Firefox were recommended browsers. 

2.2.3. Turnitin and Grademark 

Turnitin is an internet-based plagiarism-prevention service which enables submitted essays to be checked 
for unoriginal content. It normally integrates with Grademark, which enables instructors to grade students’ written 
work online by providing them the ability to add comments within the body of a paper, point out grammar and 
punctuation mistakes or works of art (Turnitin, 2015). 

One of the pertinent issues that faculty members encountered while using Turnitin and Grademark was a 
scenario where they would give feedback through commenting and pointing but students could not see the 
comments. As seemingly complicated as it looked, sometimes the issue had to do with the browser students were 
using as Turnitin, we learned, works best in Firefox and Google Chrome. Yet interestingly, in certain instances, it 
still posed varying issues with the two browsers. Apart from browser issues, sometimes the application would just 
malfunction without any reasonable diagnosable cause. These latter issues would come once in while in some sort of 
a flare. When such was the case, the Teaching and Learning Coordinator would create a ticket with Turnitin 
engineers, who would then look into the issues and mostly elucidated them by pointing out the complexity of having 
to integrate a LMS and two external applications that have to complement one another. In certain instances, faculty 
would have issues with Turnitin failing to process essays that were submitted before the application was enabled in a 
particular assignment. In such a scenario, instructors would be advised to have students resubmit the assignments 
after it (the application) had been turned on. Also, sometimes server issues would bring a glitch which resulted in 
submissions not generating an originality score due to a “class does not exist” error. Being server issues, Turnitin 
engineers would be asked to look into the problem and do necessary maintenance work. Other issues faculty 
encountered would, for example, be students’ inability to print out feedback provided through Grademark. To 
address an issue like this one, the Teaching and Learning Coordinator would devise a series of steps and walk the 
concerned faculty member through them so they could properly offer assistance to students. 

2.2.4. Respondus Lockdown Browser 

Respondus is a tool for creating and managing exams that can be printed to paper or published directly to a 
LMS. It is a custom browser that locks down the testing environment within an LMS such as Canvas. When students 
use Respondus LockDown Browser, they are unable to print, copy, go to another URL, or access other applications. 
When an assessment is started, students are locked into it until they submit it for grading. It is complimented by 
Respondus Monitor, a companion application that integrates webcam technology with LockDown Browser 
(Respondus, 2015). One of the outstanding issues that faculty would seek assistance for, was how to print Canvas 
tests using Respondus. This would require the Teaching and Learning Coordinator to walk them through steps for 
performing the action. In certain instances, users would receive an error message while trying to connect to a test 
bank server after Respondus had been enabled. Such an issue would require figuring out whether it was a Respondus 
issue or an issue to do with the owner of the test bank they were drawing questions from. Sometimes an issue would 
come up where students would report to their instructor that Respondus would not work. Such a very general issue 
required systematic diagnosis by advising the instructor to ask for more details on what was exactly happening when 
students attempted to take a test. From that information, a determination would be made, mostly engaging 
Repsondus engineers to locate the root of the problem which would be as complex as a server malfunction case. Yet 
other times, while an instructor had made all the settings correctly enabling Reposndus Monitor, students would take 
a test without being prompted to use a webcam. Again, Respondus engineers would be engaged to look into such a 
complex glitch. 

2.3. Research Question 3: What implications did these issues have on preparing future faculty development 
workshops? 
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From a general perspective, it was noted that while faculty members mostly asked general Canvas interface 
questions during workshops, more complex, context-specific issues arose when they went to use Canvas on a daily 
basis. Also, it was interesting to note that most faculty members who had reservations about Canvas at the 
beginning, preferring to continue with D2L that they had used for a long time, ended up liking it (Canvas) later after 
learning to use it. Again, with time, more and more faculty who did not teach online ended up wanting to leam 
Canvas. This group of faculty members found features like the gradebook and announcements very helpful in 
managing classes and student work. Overall, the most challenging issues arose from faculty integration of the four 
applications in Canvas. Naturally, there was variation in speed of mastery of features in the new LMS among faculty 
members. For example, some users were still learning the basics on a one-on-one basis six months after the initial 
workshops were run. Others, however, only took a couple of weeks after the workshops to get familiar with the 
whole interface. The variation in speed of mastery, it was learned, was due to different levels of proficiency with 
technology and also personal enthusiasm and motivation. 

In the long run, issues faculty members encountered helped us to go back and formatively evaluate and 
revise the workshops. Again, the whole philosophy of doing it together in instructional design (Dick, Carey, & 
Carey, 2009) became reminiscent in this study as it was clearly shown that those faculty members who preferred to 
learn by doing it together with the Teaching and Learning Coordinator easily overcame issues they were 
encountering with the technology than those who preferred for the Teaching and Learning Coordinator to solve 
problems for them. Findings of this study indicate that the whole process of implementing faculty development 
workshops require proper planning between administrators and instructional designers in order to ensure proper 
allocation of time and other resources, thereby facilitating a more efficient transition from one LMS to another. 

3. Implications 

The study presents several implications to instructional designers, faculty members, administrators and 
educational software developers on the intricate process of implementing new educational technologies and the best 
way to manage learning management system transition as a whole. First, while time is always a constraint, it is 
important to include all aspects of a LMS in a workshop in order to mitigate arising issues when faculty members 
get down to use it. It is likely that if more workshop time had been allocated to covering integration of the four 
applications to a deeper level during planning, some of the issues that arose while faculty members used them would 
have been mitigated. Also, the study does show that faculty members tend to master technological skills faster and 
more efficiently when they get involved in solving issues with instructional designers rather than having the latter do 
it for them. Again, while educational software developers do produce applications that integrate well in a LMS like 
Canvas, the study showed that there are issues that do arise when the applications are actually used in context. 

Server issues and sudden malfunction of applications as evidenced in this study provide useful feedback to the 
administrators of the four applications discussed in this study. 

In the end, the whole process of transitioning to a new LMS became an innovation being adopted. To be 
called an innovation, an idea does not have to be necessarily newly invented (Rogers, 1995; Van de Ven, 1986). As 
Rodgers (2003) contended, one of the distinct innovation attributes is complexity, the extent to which an innovation 
is considered difficult to learn and utilize. Issues that arose while faculty members used the new LMS prompted 
designers to go back and revise workshops in order to make future adoption processes of this type less difficult for 
users. Finally, the fact that this was a case study of one institution of higher learning is a limitation to this study. 
Similar studies in other institutions undergoing LMS transition including integrated applications would extend the 
scope of this study and probably collaborate the present findings. 
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AuthorlT & TutorIT: An Intelligent Tutor 
Authoring & Delivery System You Can Use 


Joseph M. Scandura PhD 
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Editor in Chief: Technology Instruction Cognition & Learning (TICL) 

I am delighted to see the growth of a now thriving international organization and appreciate Kay’s inviting 
me. Many of you have heard of adaptive learning systems — systems that adjust instruction based on learner 
progress. This paper is a short synopsis of my talk in which I introduced described and illustrated an automated 
solution that has an unprecedented chance of meeting Bloom’s standard - the effect that good tutors achieve results 
2 Sigmas better than classroom instruction. A full version of that paper will be published in Technology, 

Instruction, Cognition & Learning (TICL) journal. I should add that electronic versions of all TICL articles are now 
available to AECT members at no cost. 

TutorIT and AuthorlT build on the Structural Learning Theory - in turn on decades of basic research in 
math education, problem solving, AI, cognitive psychology, software engineering and instructional design. Under 
development for some time, they are now ready for beta use. These systems enable instructional designers to build 
dynamically adaptive (AKA intelligent) tutoring systems in their own areas of expertise. These systems also will 
enable instructional researchers to conduct more definitive research. 

My full paper begins with a short historical overview but space makes this impossible in this short 
proceeding. Suffice it to say the this research is based on a long history of basic and applied research supported by 
the US Air Force, Army, Office of Education, and NSF culminating in a $33M+ 3 year project with NIST (Dept, of 
Commerce) led by IBM. In the interim, Intelligent Tutoring systems (ITS) were introduced in the 1980s, and more 
recently, Learning Analytics - based on technical advances in BIG DATA - pioneered by Goggle. 

Although our software engineering research had nothing directly to do with education, the technologies we 
developed, including several patents, resulted in technical advances that solved a fundamental issue in knowledge 
representation in the SLT. Namely, what level of abstraction is most appropriate in representing to-be-learned 
knowledge. The short answer: We discovered a way to represent to-be-learned knowledge SIMULTANEOUSLY 
AT ALL levels of abstraction. This open a wide range of options detailed below. 

TutorIT is a dynamically adaptive tutoring system that interacts with students as would a good human tutor. 
It makes its decisions based on what individual students do and do not know at each point in time. TutorIT is 
designed to ensure mastery of the operations and decision making skills required for success - cognitive or 
otherwise. 

AuthorlT refers to a family of authoring 
platforms. We’ve use these platforms to create a 
broad variety of TutorIT tutorials. Most are now 
ready for field testing. Our immediate goal is to 
make these authoring platforms usable by non- 
programmers - to enable instructional designers 
and subject matter experts to create dynamically 
adaptive TutorIT tutorials in their own areas of 
expertise. Please see my soon to be full 
publication in TICL for relevant background. 

AuthorlT authoring and TutorIT 
delivery systems. - are based on decades of 
basic and applied research in the Structural 
Learning Theory (SLT) (cf. Scandura, 1971, 
1973, 1977, 2001, 2007). Technical 
developments based on this research over the 
past ten years are finally coming to fruition. 
Specifically, I’d like to introduce you to these recently patented technologies - most important, what they can do for 
many of you. 
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Developing dynamically adaptive (AKA “intelligent”) tutoring systems requires answers to four basic 
questions. These are questions that any good human tutor must ask. 

What do I want the student to learn? (What does it mean to know something?) Instead of worrying about how 
students learn, a good tutor needs to know what a student needs to know to be successful. This fact is the conclusion 
of my very first piece of serious research. During the heyday of the new math, a major assumption was that students 
learned better when they discovered something. Deeper analysis revealed that what really matters is what a student 
knows when he or she receives instruction. Too soon and it falls on deaf ears, too late and it didn’t matter. We later 
found that one could directly teach by exposition what was learned in discovery - and this could be done more 
efficiently (Roughead & Scandura, 1968). 

How can I find out what the student does and does not know? Tutors also need a way to determine what any 
given student does and does know — at each point in time. 

How do students acquire new knowledge? How do they learn? This question only becomes relevant once we 
know what the student must learn and what he or she already knows that is relevant. SLT assumes and data 
supports, the idea that students use higher order knowledge to acquire new knowledge (Scandura, 1967). 

How would a good human tutor put it all together to help each student learn? How do good human tutors put 
this all together in deciding what do next? Let’s take a deeper look! 

TutorIT Tutorials, based on Structural Learning Theory (SLT), offer direct answers to each of these questions: 


1. What must be learned for success: In the SLT, to 

* ? Subtract the bottom number from the top number, 
i ? REPEAT Subtract the current column. 

\ » ? Click on the current column. 

*■ ? Regroup as necessary and subtract the current column. 

• ■ ?(??) IF Is the top digit greater than or equal to the bottom digit? 

| ? THEN PREREQUISITE Subtract the bottom number from the top number. 

+■ ai5 ? ELSE Subtract the current column, regrouping as necessary. 

* ? Regroup as necessary. 

!-?(??) if can you regroup from the next column? 

* ? THEN Regroup from the next column and rename. 

\ ? Click on the column used for regrouping. 

? Regroup and rename. 

+ ? ELSE Regroup from the first non-zero top number and rename as necessa 
> ? Click on the column you should regroup from. 

: ? Regroup and rename. 

? Regroup and rename where the top number is '10 . 

| ? REPEAT Regroup and rename in the current column. 

’■?(??) UNTIL Are you ready to subtract the current column? 

? PREREQUISITE Subtract the bottom number from the top. 

: ? (??) UNTIL Have you finished subtraction? 

a high level operation on complex data . \ 


be learned knowledge is represented in terms of hierarchical 
Abstract Syntax Trees - ASTs. This screen shows how to- 
be-learned knowledge is represented in AuthorlT. This 
hierarchical representation is very important. It represents 
all possible states of knowledge about column subtraction. 
This particular hierarchy was constructed using SLT’s 
method of Structural Analysis. 

The top level node represents expert knowledge . “Subtract 
the bottom number from the top number” operates on 
subtraction problems as wholes. It represent expert 
knowledge. Given a top number and a bottom number, a 
student who has mastered column subtraction at this level 
of expertise can quickly find the difference - potentially in 
one’s head. It represents “expert knowledge”: It represents 


There are two nodes at the next level, “REPEAT Subtract the current column ” and “UNTIL Have you finished 
subtraction? The REPEAT is further refined into “Click ... ” and “Regroup ... ”. And so on until we get down to 
basic skills like “Subtract ... the current column In general, refinement continues until the skills and decisions 
required are so simple that they correspond to prerequisites that can reasonably be assumed to be available to every 
student in the target population. 


*- - Subtract the bottom number from the top number, 
f - REPEAT Subtract the current column, 
i J- + Click on the current column, 
j ■*- ? Regroup as necessary and subtract the current column. 

|- ♦ (♦+) IF Is the top digit greater than or equal to the bottom digit? 
j ♦♦+ THEN PREREQUISITE Subtract the bottom number from the top 
* a15 ? ELSE Subtract the current column, regrouping as necessary. 

* ? Regroup as necessary. 

! » ? (+?) IF Can you regroup from the next column? 
f ? THEN Regroup from the next column and rename, 
i f- + Click on the column used for regrouping, 
j : - ? Regroup and rename. 

* ? ELSE Regroup from the first non-zero top number and rename 
f ? Click on the column you should regroup from. 

: ? Regroup and rename. 

? Regroup and rename where the top number is '10 . 

| ? REPEAT Regroup and rename in the current column. 

: ? (??) until Are you ready to subtract the current column 
? PREREQUISITE Subtract the bottom number from the top. 

*• ♦ (♦+) UNTIL Have you finished subtraction? 


2. What a given student knows: As shown below, what any 
given student knows is measured relative to the initial 
knowledge representation. Individual nodes are marked “+” 
(mastered), (known to be unknown and “?” (status yet to be 
determined). The knowledge representation for this particular 
student, for example, at this particular point in time, is known to 
have mastered the nodes in green preceded by a “+” sign. 

Those is red preceded by a sign are known to be unknown. 
Notice also that the top level red node in the hierarchy above the 
“ REPEAT Subtract ... ” node is also is red and marked 
Conversely all nodes below any “+” green node will necessarily 
be “+”. Prerequisite nodes are at the bottom of the hierarchy. 

The way this inferencing works is very important. The 
various transitions are illustrated in the schematic below. The 
initial hierarchy on the left represents a starting point. The 
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learning status of the student at this point in time is assumed to be completely unknown. Hence, the student’s 
knowledge is unknown to TutorlT. This is illustrated in the figure to the left (with all question “?” marks) in the 
elements. The center figure signifies that the student has demonstrated mastery of tasks associated with the green 
highlighted node with a “+” sign. Observing this, TutorlT automatically infers that the student also knows the 
prerequisite nodes below as shown in the figure at the right. In this case, all are made green and marked “+”. 


? 



■ — 1 — 

1 


a — r — 


+ 

? 

j 1 

? 

+ 

? 

? 




higher level nodes (in this case only the top level node) with a 



The figures to the left show what happens 
when the student fails on tasks associated a node 
in the knowledge representation. This is shown 
by the red node marked in the figure on the 
left. Analogous to the inferencing used in the 
case of success (+), TutorlT in this case also 
marks 

- designating unknown. 


3. How Students Use What they Know to Acquire New knowledge. It is beyond the scope of this paper to detail 
the precise mechanisms used in SLT to explain the learning process. For specifics, please see Scandura (cf. 1971, 
1973, 2007). 



Second AST 




One AST Second AST 





Each of these arrows represents a 
hierarchy. Think of it as a “chunk” of to-be- 
learned knowledge. The higher order AST 
“chunk” in the middle operates on AST chunks and 
generates new ones (e.g., the combined chunk at 
the bottom). In the SLT (Structural Learning 
Theory) students learn or acquire new knowledge 
using their higher order knowledge to create new 
knowledge from their existing knowledge. Lower 
and hlhger order ASTs (SLT rules) are both 
derived via Structural Analysis (SA) from the 
given content domain. 

4. How TutorlT Works. The following figure 
puts it all together. The Knowledge Representation 
at the top represents what students are to learn. 
TutorlT interacts with the Learner (student) 
through the Blackboard Interface. TutorlT displays 
problems and/or partially solved problems on the 
Blackboard Interface. The Learner responds. 
TutorlT evaluates each Learner’s response and 
reacts accordingly. 

As above, paralleling what a human tutor 
might do, TutorlT marks nodes in each hierarchical 
AST display as known or unknown as the case may 
be. Furthermore, hierarchical relationships 
between the nodes allow TutorlT to infer status on 

untested nodes . This is essentially what human 
tutors do instinctively - this type of inferencing is 
what makes it possible for human tutors to quickly 
home in on student needs. The main difference is 
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that TutorlT does this both automatically and more systematically . 


There is obviously a lot more to say about how Learners’ learn, how TutorlT makes its decisions and the 
relationships between them. As we shall see, there also is a lot to say about our authoring processes. Toward this 
end, I recommend that you review the articles referenced under “GET AuthorlT/TutorIT Publications” at 
www.TutorlTweb.com . Key ideas are unique and recently patented. 


Sample TutorlT tutorials.— Following are sample entry and introductory screens providing an overview. 



The video links are optional. They are used to provide either overviews 

and/or introductions to the tutorial. The first is a Flash file. 
The second, a YouTube video that provides an introduction 
to Column Subtraction. 


In the screen shot on the left, TutorlT is essentially asking 
the student where to begin Column Subtract. The one on 
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learner model on the right shows the status of a more advanced student. Notice that this student hasn’t made any 
mistakes - likely because he or she has already been exposed to column subtraction. 

The screen shot on the left shows what happens when the student makes a mistake — TutorIT immediately shows 
what was done wrong, and provides instruction on what the student should have done. 


TutorIT Showing Full Mastery 


Learner Model 



- 6 


0 7 

4 9 


ora* i 


' 20 


The 
one 
above 
on the 
right 
shows 
what 
the 

student 
has 

done to demonstrate full mastery. TutorIT continues 
tutoring until a student demonstrates mastery of each and every operation and decision that the author believes 
students should know about column subtraction. Like a human tutor, TutorIT will continue tutoring until the student 

either demonstrates mastery or gives up. This is what we 
mean by guaranteed learning. 

TutorIT need not stop just because a student has demonstrated 
mastery. Some solution methods, or parts thereof, may be 
judged to be so important that an author may require students 
to perform various combinations of steps in their head -- even 
after the student has demonstrated mastery. At this point, the 
student has demonstrated mastery of the full column 
subtraction process, and the author decided to require the 
student do all the regrouping et al in his or her head - in this 
case with 15 second (see the arrow). 





VO- 



The two below on signed numbers 
require the student to enter two 
kinds of correct answer, one 
numerical, the other by clicking 
on the number line. 


“K 
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+3 x -6 = 


♦3 x -6 = -18 
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subject. One on the left involves critical reading. 


The tutorial on the next page involves solving quadratic 
equations. Here, the student can choose to solve the 
problem in any of three ways - factoring on the left, 
using the quadratic equation on the right and completing 
the square in the middle. 

Tutorials need not be in mathematics, or any other 
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The screen shots on the right below shows 
two other TutorIT tutorials. These tutorials 
are presently samples but give a broader sense 
of what can be done. The one the left shows 


the Periodic Table in Chemistry. Students can explore properties of various elements before answering questions. 
The other screen shots are from a sample TutorIT tutorial preparing lawyers for the law boards. The law boards are 
required to practice in most states. If an aspiring lawyer gets a question correct, he is congratulated and told why his 
answer is correct. If incorrect, he is told why. 

Currently available TutorlTweb tutorials include: Basic Facts, Whole Number Algorithms, Fractions, Signed 
Numbers, Complex Expressions, Math processes, Critical Reading/Logical Analysis, Simplifying Algebraic 
Expressions, Solving Linear Equations, Solving Simultaneous Linear Equations and Solving Quadratic Equations. 
Samples include: Word Problems, SAT Prep, Law Boards and the Periodic Table. 


Beta Testing and Classroom Use.— Most of these systems are now ready for beta testing . 1 TutorIT tutorials 
obviously can be used by individual students. Another use is perhaps less obvious - precisely because TutorIT 
decision making as well as level of detail is so unique. TutorIT tutorials are not limited to simply presenting 
problems and/or questions, evaluating student responses and adapting future instruction. TutorIT tutorials pinpoint 
the cognitive processes and decision making skills students must learn to be successful - at each point in time. At 
each stage, TutorIT presents precisely the information needed by the student to progress. The process continues 
until the student demonstrates full mastery. Irrespective of whether a teacher or instructional designer agrees with 
all aspects of the solution method being taught, a good deal of thought has gone into each TutorIT tutorial, generally 
speaking considerably more than most teachers have time to do. 

Accordingly, TutorIT tutorials can be thought of (and used) as carefully analyzed lesson plans. At each stage of 
learning, TutorIT will do automatically what a good teacher might do. TutorIT will present questions and partially 
solved problems — systematically step by step. TutorIT will ask students for answers, grade those answers and 
provide precisely the help needed where needed. The process will continue as long as the teacher desires, or until 
the class demonstrates full mastery. In the same way that TutorIT tutorials may be supplemented with YouTube 
videos or other media, the teacher also may intervene during the course of instruction with whatever additional or 
supplemental information may be desired. 


In short, TutorIT tutorials can be viewed and used as automated lesson plans. We plan to make TutorIT 
tutorials available for teacher in class at no cost. Teachers, and instruction designers working with teachers, are 
encouraged to contact us for further information. Perhaps surprisingly, TutorIT does this more systematically than 
most teachers typically do in class . 2 


1 Those interested in evaluating these tutorials are encouraged to contact us at scandura@scandura.com. 

2 For testing purposes, and to reduce the load of these sophisticated systems on our server, we initially are limiting 
use of TutorIT tutorials for teacher use in intact classrooms. 
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TutorIT Authoring Systems,— The following figure provides an overview of the authoring process. Focusing on 
the left side shows that Structural Analysis (SA) is used to represent to-be-acquired knowledge in a form that can be 
used by TutorIT. Our AuthorIT authoring tools make it possible for authors to create TutorIT tutorials of increasing 



sophistication and scope. In each case, the author must create a representation of the to-be-learned knowledge (the 
Tutor content), show how problems are to be laid out on the Blackboard interface and specify how the content is to 
be delivered. Unlike any other authoring system, the same content may be delivered in many different ways, by 
simply selecting options in a dialog. For example, the same content may serve as the foundation for a dynamically 
adaptive (AKA intelligent) tutoring system, as an adaptive diagnostic test, as simple test or simple performance aid 
directing a student or trainee step by step in how to perform some arbitrarily complex task. Even more surprising, 
recently patented methods enable TutorIT to perform all of this decision making automatically - without any 
programming whatsoever. 



Authoring: Create/Edit TutorIT Content 

1 . Select an authoring tool. 

,. J 

2. Select the content to author. If creating new content, select 'New". New content will be added to 
your Custom Content directory. 

To make re-* content part of tre ofWal wee content, It meet ce aooeo to ire AKorlertkW me 
Tre nffl tie it racaieo in e flpfoor^ trc'iopiTutoriTwefi'aata’AliContenUimi 
Follow me instructions kitUe tne xml me wren aoomg. 


1 1 002Jms v | [Select Pac kage ~ 


*] Select SkiH v 


Install AuthorIT Desktop 


These screen shots show that 
four authoring systems are 
available for constructing or 
customizing TutorIT tutorials, 
and for integrating same. 

There are three increasingly 
sophisticated TutorIT authoring 
systems: EZauthor, Customizer 
and AuthorIT (itself) plus a 
Scope & Sequence tool for 
putting them together to create 
entire curricula. 
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EZauthor 

EZauthor is designed for use 
with unstructured content — 
like the SAT, foreign 
language vocabulary, basic 
facts, etc. Most educators, 
including busy teachers can 
learn to use EZauthor in a 
very short time. By the time 
this goes to press, It will be 
possible for instructional 
designers, curriculum 
developers, trainers, tutoring 
center owners to create their 
own TutorIT tutorials for 
use by their students, and 
also to submit them for 
broader use. 


m AuthorIT & TutorIT 

Dyfvwmcjify Aoapti.. Tutoring to* Evoryon* (DATE)' 

Horn* AuthorIT TutorIT Monag* Monaga ^jbllc«tton« Logo u t 



Click to ask the student a Question. 


Click to enter the 
information on w 
answer is to be t 


Click to enter the 
answer. 


Click to enter coRrective feedback for wrong answers. 


T 


EZauthor Properties 


Rename Cop> Delete 
Intro/Rnal Screens Tutoring Screen Options 


Default cattr 


Select Picture 


e Duplicate v 

■*=“ 51 -: otangeooer 


5-neTeprooiemor 


fc 150 top: 175 width: W height: S3 


m much re student 


** Hide Guidance Hide Properties 




Advanced View Test Content Save Quit 


Customizer 


Customizer also is very easy to use. Any instructional designer can easily customize any of the professionally 

developed TutorIT 

■n m in 



i 
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You rod tuWKt eo ur by ceont i 


Good l You a^ now apo to perform coamn a 


UNTu Have you 


tutorials currently 
available. For 
example, it is easy to 
adjust the wording of 
Instruction, 

Questions, Feedback 
or Corrective 
feedback. One can 
even add one’s own 
voice by loading in a 
recording or a new 
language (e.g., using 
Text-to-Speech). 
Customizer also 
enables authors to add 
one’s favorite 
YouTube videos, or 
other media. 


I 

* 

I 



Prppar# For T#vtmp 


Tost content Sava 


I 


AuthorIT 

Whereas Customizer works (and customizes) professionally developed TutorIT tutorials, Author itself goes much 
further. Given any content domain, AuthorIT both guides and empowers authors to perform arbitrarily detailed 
Structural Analysis to represent the knowledge or skill to be acquired. In the case of structured knowledge, 
knowledge is cumulative in nature (as in column subtraction, solving quadratic equations, etc.). This is illustrated 
below in the knowledge representation shown on the left. 
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In short, AuthorIT is a full featured professional 
authoring system. AuthorIT is used to create 
hierarchical knowledge representations for 
essentially any content (see above). It also is used 
to layout problems on the Blackboard through 
which TutorIT interacts with students. The 
screen shot below shows the blackboard itself. 
Problems (or problem schemas) are laid out in the 
center frame. This “Blackboard” interface (not 
the Blackboard LMS) represents the interface 
through which TutorIT and students are to 
interact. The left side represents the problem 
structure. To its right are properties of each 
problems element. These are used by TutorIT in 
deciding how to interact with students. 

AuthorIT is designed for use by authors with 
some degree of technical experience. 

Professional instructional designers and other 
subject matter experts can use AuthorIT to create 
'W , ■ t B dynamically adaptive TutorIT tutorials with all of 

the features previously described. 

“ - ■ ■ “ The desktop version of AuthorIT was used to 

build our existing TutorIT tutorials. It is a powerful full featured authoring system — but requires some degree of 
technical sophistication. The screen on the right shows the beginnings of an easier to use web based version 
currently under development. 
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Scope & Sequence 

The Scope & 
Sequence authoring 
system is used to 
define entire curricula 
composed of existing 
TutorIT tutorials. The 
author simply selects 
and specifies 
hierarchical 
relationships between 
available TutorIT 
tutorials. This is done 
irrespective of how 
they were created - 
with EZauthor, 
Customizer or 
AuthorIT. 

Scope & Sequence 

hierarchies may be run as either diagnostic or curriculum mode. Running in Diagnostic mode is used to determine 
where in a curriculum, any given student should begin (and progress). Run in curriculum mode will automatically 
take a student thorough and entire integrated courses or curricula. What makes Scope & Sequence tool particularly 
useful is that it is particularly easy to create custom Scopes & Sequences, made up of “official” and/or custom 
TutorIT tutorials. Run a Scope & Sequence as a diagnostic, TutorIT quickly determines which Tutorials a student 
has mastered and which not. Run in curriculum mode, TutorIT provides a general road map for entire curricula, 
ensuring that the student masters each tutorial in a curriculum before moving to the next. In all cases, TutorIT 
makes inferences about what students do and do not know, ensuring optimal efficiency in the learning process. 

Each student gets precisely the tutoring needed to progress in optimal fashion - moving ahead at the rate required to 
ensure prescribed levels of mastery throughout the process. 
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Managing Delivery Modes 

AuthorIT also includes a broad variety of management options. These options enable teachers to customize delivery 
of individual tutorials. TutorIT delivery options, for example, include automatically enabling any given tutorial to 
serve as an efficient diagnostic engine or as a dynamically adaptive tutorial. These options also enable researchers 
to compare alternative instructional methods using the same content with unprecedented precision. Such research, 
for example, could help put to rest such age old questions as to whether and why it is better for students to discover 
knowledge on their own rather than being told. 

TICL On-Line.-- Last but not least, researchers courtesy of Old City Publishing (formerly Gordon & Breach 
Science Publishers) now have Free Access to ALL articles published in Technology, Instruction, Cognition & 
Learning (TICL). Log in directly at www.TutorlTweb.com or from the AECT website. 


Concluding Statements and Opportunities.— Interested readers may contact us for specifics at 
scandura@scandura.com: 

(1) If interested in field testing TutorIT, please send us a short description of planned research. Please inc lude 

content area, research interests and whether 
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your research is funded or unfunded. How 
do you think TutorIT or AuthorIT would 
help? 

(2) If interested in developing TutorIT 
tutorials in your own areas of expertise, 
please send a short description of the 
TutorIT tutorial you would like to create. 
Please include how you plan to use the 
tutorials and your preference as to authoring 
tool : EZauthor, Customizer, AuthorIT or 
Scope& Sequence? Please also include an 
assessment of your (or team’s) computer 
skills (e.g., tool familiarity??? Any 
programming skills, etc.) 
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Abstract 

Learning objects (LOs) are good tools for instructional design. They are easy to be modified once 
instruction is complete. Many authors have approached them differently. The counterparts of LOs in the field of 
software engineering are called objects (Alonso, Lopez, Manrique & Vines, 2008), which are used as a reference 
point in this paper. Object oriented programming (OOP) and unified modeling language (UML) has dominated 
software engineering industry in the past few decades. UML has helped software architects visualize a software 
before it is actually developed, thus saving time and money. No similar modeling language exists in the field of 
educational technology. Most of the instructional design is done through editing lengthy text over many iterations. 

A new instructional modeling language (IML) designed by the author is introduced in this paper for the first time. A 
parallel has been drawn between objects in software engineering with LOs in the field of instructional design. The 
LOs’ collection could be extended to build a repository which is shared among educators. The key element of LOs 
is re -usability which has been explored. This paper explains the similarity that exists between objects and LOs with 
the introduction of a new modeling language (IML). It also highlights the advantages and challenges that they both 
offer. 

Keywords: objects, learning objects, unified modeling language 

Introduction 

Dowens (2004) defines learning objects in instructional design as anything that can be used, re-used or 
referenced in technology. On the other hand, an object in software engineering is defined as something that has 
attributes and behaviors (Rumbaugh, Jacobson, & Booch, 1999). Objects and object oriented programming (OOP) 
have revolutionized the software engineering industry in terms of their popularity and re -usability. Most object 
oriented languages (OOL) responded to the idea of OOP by matching language constructs to the concepts that 
revolved around objects. Many object oriented design (OOD) notations like UML were introduced that helped in 
designing software systems. However, no such notational language exists for LOs. 

The goal of this paper is not merely to draw a similarity between objects and LOs. Instead, a rigorous 
methodology has been suggested to use a similar approach which had been used in software engineering for 
decades. This approach will benefit instructional design from its concept, design, implementation, and re -usability. 
LOs can even become of greater importance when used in their repositories. This is where the power of re -usability 
can be seen in action. Many topics related to OOP are re-visited to see where we can benefit from its concept and 
implementation. There is a lot that could be done in the field of instructional design with the help of LOs. 
Instructional modeling language (IML) has been introduced for the first time in this paper which could simplify the 
process of designing and launching a course, documenting requirements, and bringing instructional designers, 
subject matter experts, and software engineers on the same page. The field of instructional design is going through 
the same process of reformation as we saw the changes in the field of software engineering a few decades ago 
through OOP. 

Literature Review 

Before going into the advantages that OOP provides in software engineering, a few terms are defined and 
the similarity that exists between OOP and instructional design is described. Once the reader understands what an 
object is, it is logical to then think about a class. A class is a group of objects (Rumbaugh, Jacobson, & Booch, 
1999). For example, “human being” is a class and “John Doe” is an object. In instructional design, the counterpart 
of a class is a learning objective (LJ). LJs are defined as a set of LOs that can be evaluated according to 
performance goals in order to develop coherent information structures that help knowledge schemata in a learner’s 
mind (Donavan, Bransford, & Pellegrino, 1999). 

OOP is done through OOL. Some of the OOL include Java, C#, Python, etc. The first OOL developed is 
generally acknowledged to be Simula-67 in 1967. However, the concept did not gain popularity until the 1980s and 
1990s when some of the later languages such as “Smalltalk”, “objective C”, “C++”, etc. appeared (Rumbaugh, 
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Jacobson, & Booch, 1999). An OOL must have certain characteristics, such as encapsulation, inheritance, 
polymorphism, and dynamic binding in order for it to be considered an OOL (Craig, 2007); however, these concepts 
are taught in software engineering and are beyond the scope of this paper. For example, “C” programming language 
is not an OOL because it does not have all of the mentioned characteristics. OOL was a new concept compared to 
the traditional procedural languages like “C”. 

UML is a modeling language used by a developer, architect, project manager, system engineer, 
programmer, analyst, contracting officer, customer, and anyone else who wants to design, build, and understand 
complex software systems (Rumbaugh, Jacobson, & Booch, 1999). UML is used to model objects. It provides 
concepts to draw various diagrams involving objects, classes, activities, etc. which depict a system from various 
perspectives. Conceptual models enhance conceptual learning (Bruner, Goodnow, & Austin, 1967). A deeper 
understanding of the requirements about objects is necessary by a software engineer before they could start 
programming in an OOL. This understanding is enhanced by using an UML model. This paper introduces a 
counterpart of UML called IML for the first time. Instead of using graphical notations as suggested by UML for 
every concept in software engineering, LOs are abbreviated through acronyms in the IML proposed by the author in 
this paper. 

The main characteristics of objects as well as their usage are very important in OOP. For example, re-using 
an object again-and-again saves time and thus cost. Before the concept of an object, programming was done using 
procedural languages like “C”, “Pascal”, etc. Those languages had data and procedures spread all over the code. 
OOP gave software engineers a natural way of looking at a problem domain as objects and thus programming in it. 
Almost everything that we see is an object. OOP gave an easy to design the software system approach. One of the 
main advantages of OOP is reusability. This reusability saves time which ultimately leads to cost savings. For 
example, the paper that you are reading is an object. Following our definition above, the paper has attributes, such 
as, topic, text, color, language, and so forth. It has behaviors or functions, such as, read, save, cite, etc. Another 
example of an object could be an examination in a course (e.g. MATH 101) which has attributes like questions, 
grading, method of delivery, and so forth. 

Based on our definition of an LJ which is made of many LOs, a syllabus could be used as a counterpart of a 
class. LOs could also be used and re-used (whether using a technology or not) which fits the definition. For a 
different course’s syllabus, we would probably change the name of the instructor, location, and so forth but most of 
the contents would stay the same. LJ is discussed with greater detail in this paper. Each of the items included in a 
syllabus, such as course description, grading policies, examination, homework, etc. are objects or LOs. 
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Methodology 


The definition of a class serves as a template which consists of attributes and behaviors. An object is an 
instance of a class and thus has all the attributes and behaviors of a class. While this is the definition in software 
engineering and it fits the domain, instructional designers do not deal with behaviors. On top of this, the goal of this 
paper is not try to teach software engineering to the instructional designers. They are two different disciplines. Both 
disciplines can benefit from each other. Consider our example of the course MATH 101. If we design another 
course (e.g. ENG 101) that has everything that MATH 101 offers except different values for course description, 
grading policies, method of delivery, location, etc. - then the new course (ENG 101) is another object of the class 
“Course”. The class and object relationship in software engineering can be illustrated by using UML in Figure 1. 

Class 

4 



Figure 1. Software engineering approach for class (Course) and objects (MATH and ENG) using UML 

Only two attributes are listed in Figure 1. Some of the other attributes could also be included such as: method of 
delivery, location, and so forth. Similarly, only one behavior (Add Course) is shown in the class course. Some of the 
other behaviors or functions that can be added to the class “Course” are “Drop Course”, “TakeExamination”, 
“DoHomework”, etc. There are only two objects named “MATH” and “ENG” which are shown in the Figure 1. More 
objects are added on as needed basis. The above example could be defined in the proposed instructional modeling 
language as follows: a class in Figure 1 is replaced by a new term; “super learning objective” (SLJ). The objects of Figure 
1 are replaced by “learning objectivel” (LJ1) and “learning objective2” (LJ2), respectively. In other words, we are 
redefining the class as “super learning objective” (SLJ) instead of LJ only. However, our SLJ could have as many LJs as 
needed. There is no instance of a class in instructional design as in Figure 1 for software engineering. However, we have 
LOs in instructional design. This way, we have preserved the main concepts of instructional design without a compromise. 
One might ask about the advantages of using UML and/or IML. The answer can be traced back when UML was 
introduced for software engineers. There was no modeling notation before UML in software engineering. Many authors 
were using their own graphical notation to illustrate objects and classes. UML has provided the standard that is used 
industry wide for software engineering. This standard eliminates a lot of confusion and facilitates understanding of 
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objects. A diagram drawn in UML similar to Figure 1 could be then assigned to software developers who would use it in 
a requirement or a design document and write code in programming languages like “Java”. In Figure 1, we just needed the 
class to provide the template only. The real work is done in the objects when doing OOP by the programmers or software 
engineers. The concept of class and the objects from Figure 1 are replaced in our proposed IML as SLJ, LJ1, and LJ2 
while preserving their definitions as they relate to instructional design. The LOs are part of SLJ or LJ in our IML and are 
shown in Figure 2. 

SLJ 


Name 

LOl 

L03 

L05 

More LOs 


LOs 

> 


Y 



Figure 2: Learning Objects using Instructional Design Approach with description at the bottom about LJ1 and LJ2 


There seems to be a universal agreement when it comes to LOs and the aspect of re -usability (Tono, & Lee, 
2011). That means we can use the same template SLJ (Freshman Courses) from Figure 2 for as many courses as we wish. 
The thing to remember in our design is that SLJ must have all the possibilities of LOs that an LJ (course) could possibly 
have. However, the LJs do not have to have all the LOs from the list of SLJ. This is another place where we distinguish 
instructional design from software engineering. In Figure 2, the SLJ has “Online Quizzes” but LJ2 (ENG 101) does not. 

So, in our model it works a little different than inheritance and objects in OOP. In other words, SLJ serves as a template 
from which LJs could borrow LOs on as needed basis. The value of the LOs will change but the template stays the same. 
In Figure 2, the two values of LOs changed to “MATH 101” and “ENG 101” but the template (“Course Number”) stayed 
the same. This saves time and could be used as standardization of how we structure courses. Another advantage of using 
these definitions is giving it the software engineering perspective so software engineers can build a better learning 
management systems (LMS) by understanding the IML. Most of the software engineers are comfortable with UML and 
therefore they would not have any problem in understanding IML. Today’s latest learning management systems (LMSs) 
are built using OOL. Figure 1 and Figure 2 could be used to complement each other as part of the design document in 
software engineering. This will help software developers understand the requirements better. At the same time, 
instructional designers do not have to leam the details of OOP which could be challenging. Most of the software engineers 
will have no difficulty in converting SLJs, LJs, and LOs to classes and objects and then programming them to build LMSs. 
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We have already introduced IML in Figure 2; however let us ponder upon all the possibilities of a course’s 
content. The research done in this paper can be extended from basic course content to a very sophisticated course offered 
through mobile devices. The current literature does not provide a conceptual model for applications for handheld 
technology (Churchill, 2011). Almost all courses have a syllabus which could contain a plethora of contents such as 
course description, homework, quizzes, exams, class participation, grading policy, textbooks, reading list (books), articles 
to read, online contents, labs, lab instructions, reference books, instructor’s office info., instructor’s contact info., 
instructor’s picture, discussions, journals, individual projects, group projects, videos and online submission information, 
and so forth. The recommendation to add a description in the definition of LO from Sosteric and Heseimer (2004) makes 
sense. Therefore, description has been added as part of our IML as well. Instead of reading long texts of design 
information or designing in LMS and editing it over and over again where one could get lost and forget most of the 
information, a simple and easy to remember graphical notation is introduced in IML. That means, a course is designed 
using IML and then text can be put appropriately using LMS. This design could also be beneficial to software engineers 
before they design and program an LMS. The notations proposed for IML are as follows: 


^ S L J = Super Learning Objective = e.g., freshman 


Courses. 

> LOl, L02, and so forth are the Learning Objects. 


^ This is the description part of the LO 


S L J (Freshman 
Courses) 

LOl 

L02 

This is S L J for Freshman 
Math Courses 


Figured : Super Learning Objective (SLJ) and Los 

In Figure 3, SLJ is represented by a rectangle divided by two horizontal lines. The top area is used for the name 
of the SLJ, the middle area is used to list all the LOs, and the bottom area is for the description about the SLJ. There is no 
difference between the notation for SLJ and LJ. That means both are represented by a rectangle in IML. So, SLJ will have 
its’ type within the parenthesis, e.g., SLJ (Freshman Courses) whereas, LJ will be written according to the actual value of 
the LJ’s course which in our example is Math 101 and it is one type of freshman course. We do not want to restrict the 
words in parenthesis so that instructional designers could put whatever is self-explanatory. An example of an LJ which 
uses a similar template as SLJ could be drawn as follows in Figure 4 for Math 101 : 


^ Learning Objective (Math 101) 


^ Learning Objects (LOs) and their acronyms 


Figure 4\ Learning Objective (L J) for Math 101 
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The goal of our graphical notation is to multi-fold, be simple to understand and use, to avoid the textual 
details, and to create a standard for the future design of instruction. Instead of introducing too many graphical 
notations, we are limiting to whatever is needed. Similarly, we are not designing more graphical notations for LOs. 
Rather, easy-to-remember and self-explanatory acronyms are used which are explained below. 

Acronyms for LOs 

The following acronyms are used in our IML. 

LO - Learning object 

LJ - Learning objective 

LJn - Learning objective number (e.g. LJ1) 

SLJ - Super learning objective 

LJm - Learning objective through mobile devices 

LO nl - LO for course number 

LO_n2 - LO for course name 

LO si - LO for school information 

LO nn - Any LO that starts with the letter n for the future 

LO cd - LO for course description 

LO cg - LO for course goals 

LO ce - LO for course expectations 

LO_e - LO for examination 

LO en - LO for examination number such as 1, 2, 3, etc. 

LO ef - LO for final examination 

LO em - LO for midterm examination 

LO_h - LO for home works 

LO hn - LO for home works’s number such as 1, 2, 3, etc. 

LO_q - LO for quizzes 

LO qn - LO for quiz number such as 1, 2, 3, etc. 

LO oq - LO for online quiz 

LO oqn - LO for online quiz number such as 1, 2, 3, etc. 

LO cp - LO for class participation 

LO gp - LO for grading policy 

LO gs - LO for grading scale 

LO sp - LO for school’s policies 

LO cm - LO for course materials 

LO_o - LO for course outline 

LO tb - LO for textbook 

LO rl - LO for reading list (books) 

LO ra - LO for reading list (articles) 

LO oc - LO for reading list (online content) 

LO rb - LO for reference books 


330 



LOl 

LO for labs 

LOln 

LO for labs number such as 1,2, 3, etc. 

LOli 

LO for lab instructions 

LOio 

LO for instructor office information (location) 

LOip 

LO for instructor’s picture 

LO_d 

LO for discussions 

LOde 

LO for fescriptions 

LOJ 

LO for journals 

LOs 

LO for schedule 

LOip 

LO for individual projects 

LOgp 

LO for group projects 

LOos 

LO for online submission instructions 

LOljde 

LO for description about LJ 

LOsljde 

LO for description about SLJ 


Please note that numbers could be added for any type of design if needed. For example, if there are two 
individual projects (ip), the IML will be LO ipl and LO_ip2 respectively. This flexibility is always available to the 
instructional designers who are using the IML. 

The suggestion given by Sosteric and Heseimer (2004) about adding a description to a learning object is a 
good idea and therefore we have included a description with class (LJ) as LOljde. This description explains the 
details about the LJ. In the Figures 2 through 5(below), description is shown at the bottom of the rectangle for SLJ 
and LJ. The size of these rectangles could grow depending upon the number of LOs and the length of the 
description. Every LO can have a separate description of itself in a textual format. This description will make a 
design more understandable to the reader. However, a separate appendix should be used to add the descriptions 
about all of the LOs, if needed. Let us re-visit the definition of LJ and LOs from above. For example, LOl could be 
examl and L02 could be another exam2 as shown below or homework and so forth that are offered in a course. 

LJ = MATH 101 

LOnl = Math 101 

LO_n2 = Intro to Math 

LOel = Examl 

LO_e2 = Exam 2 

LO_h = Home work 

LO oq = Online Quiz 

LO de = This Math class is used for freshman 


Math 101 
Math 101 
Intro to Math 
Examl - 30% 
Exam2 - 30% 
Home work - 20% 
Online Quiz - 20% 

This Math class is 
used for Freshman 


Figure5 : Sample LJ is shown with actual values for the Figure 4 above 
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It should be noted that the size of this notation (rectangle) does not matter. However, what matters is the 
shape (must be a rectangle for SLJ and LJ) and the description of the language, IML. The word “class” and course 
should not be mixed in this paper. The words “class” and “object” are dedicated for the discussion of LJ and LOs 
respectively and are borrowed from the field of software engineering. The process of designing a course can be 
accomplished with the help of SLJ, LJ and LOs. As we have seen above, writing of lengthy texts is not necessary 
because it could take forever to read and is confusing. IML could be taught and used for a better instructional 
design. 


Repositories 

There is an acute need for the development of an instructional design tool and sharing of educational 
material which would help faculty in the designing of curriculum and expand the knowledge of students and faculty 
alike. No standardized tool exists today in the academia to design a curriculum or share educational material without 
long textual explanations which take a lot of the educators’ time and resources. Present learning management 
systems (LMSs) depend on the time consuming process of writing, selecting, editing, and posting complex contents 
for a curriculum. IML can be used by faculty of all disciplines at all levels of education for the designing of a 
curriculum, LOs, and their relationships. IML promises to provide flexibility in designing a course in a systematic 
fashion while defining a new standard. IML can be expanded and coupled with a repository of learning objects 
(LOs) or existing LMSs. IML promises a new tool that is complete, easy to use, and time saving for faculty. It also 
provides an opportunity to the teachers to share knowledge on an everyday basis. More importantly, it will bring the 
educational technology community to agree upon a single definition of LOs. There has been a long discussion over 
the years for the standardizing LOs without a compromise. The LOs’ repositories can be uploaded on an Intranet 
and shared among the teachers and students within an organization and across organizations. These repositories can 
be made available to the teachers and students 24/7 throughout the year. Repositories stored electronically are the 
best way to share useful information among educators and students alike (Carrion, Gordo, & Sanchez- Alonso, 

2007). Currier, Barton, O’Ceirne, & Ryan (2004) described about the quality of LOs as it relates to metadata. 
Specifically, they researched about the creation and the quality of metadata repositories. They acknowledged the 
importance of metadata and their use over the web. However, they argued that the creation of the repositories had 
been overlooked. LOs could be a perfect choice to design metadata repositories and store it on an intranet to be 
shared by the knowledge seekers of all levels and profession. 

Limitations 

The only drawback from instructional designers’ perspective is that they might not know the original 
concepts of OOP and thus will take some time to understand graphical notations like IML. A simple training about 
IML should suffice. The IML does need a feasibility study before it can be adopted in the academia. Most 
instructional designers use ready-made LMSs to design a course. However, most of these designs are either done on 
the fly or by providing lengthy textual descriptions making it a time consuming process. A notational language like 
IML might be confusing at first, however, it is needed to design courses in the field of instructional design. This 
will allow the instructional designers to visualize the course before it is built in LMS. Instead of going through long 
texts of information which could be confusing, a graphical notation like IML might be the answer to many of the 
intricacies that the field of instructional design offers. On top of that, graphical notations and acronyms are much 
easier to learn and remember. Hence, the benefits outweigh the limitations. 

Future Research 

LOs offer hope for future research as well. Since the concept of LOs was borrowed from software 
engineering, not every concept from software engineering has been explored. For example, the concepts of 
encapsulation, inheritance, and polymorphism could be looked into in finding any benefit which might exist in order 
to introduce their counterparts in instructional design. LOs could further be explored as it relates to the 
implementation of human performance technology. As more sophisticated software is developed to enhance the 
power of instructional design, more avenues will open as it relates to LOs and IML. IML could be distributed to 
teachers at all levels for a prototype testing. This data can finally be tested using inferential statistics (Creswell, 
2012) which would determine the feasibility of IML in the field of educational technology. There is a lot of work 
which needs to take place to design LOs’ repositories, and IML can play a very important role. 

Conclusion 
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The research in this paper showed that the idea behind UML revolutionized the development of software. 
While many instructors struggle and spend a lot of time in designing a good course, IML might be the answer to the 
instructional designers. If implemented with the proper notation, it can simplify the process. This is in alignment 
with software engineering where classes/objects are defined in terms of abstractions that are composed of data and 
behaviors. These abstractions are more useful if they are made digital and implemented in one of the object oriented 
programming languages such as “Java” or “C#” (pronounced as C-Sharp). OOP also makes use of other concepts 
such as inheritance, polymorphism, and dynamic binding. Those concepts work well and are widely used in software 
engineering. However, this paper did not apply every aspect of software engineering into instructional design. This 
is an open research area. LOs repositories could be an excellent source of knowledge and sharing among teachers of 
all levels. This will save cost and give more time to teachers to work on other much needed areas. IML can be used 
without a lot of complexity and compromise for the two disciplines; instructional design and software engineering. 
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Abstract 

Instructional design novices learn to identify and address performance problems by studying authentic 
cases. An authentic case was created for a face-to-face setting to demonstrate to graduate students the complexities 
of the ID process. The case is interactive, eliciting frequent responses and reflections, and progressive, requiring 
students to adjust their assumptions, opinions, and recommendations based on emerging information. 

Introduction 

The design of a low-tech, face-to-face, interactive, progressive case to train novice instructional design (ID) 
learners about the complexities of ID processes is outlined. The case is interactive, eliciting frequent reflections and 
responses, and progressive, requiring learners to adjust assumptions, opinions, and recommendations based on 
emerging information. 

Context/Problem 

Learners in this introductory Instructional Design course consist of first-year Master’s students who often have 
difficulty with: 

1 . Identifying and stating performance problems 

2. Considering instructional and non-instmctional solutions 

3. Recognizing that the ID process is iterative and not linear 

These students often hesitate to actively participate in traditional lecture-type sessions, however tend to 
raise questions during short activity debriefs conducted in classes. This often causes disconnects between ID 
theories and practices. 

To address learner needs, case studies were gradually added to the course to bridge the gap between theory 
and practice and prompt “novices to practice ‘thinking like’ professionals” (Ertmer & Russell, 1995, p. 25). Case 
studies have been found to be beneficial in training instructional designers (Ertmer & Russell, 1995; Sugar, 2014). 
Learners reported that case studies were “valuable in helping them synthesize course content. . .solve real world 
problems, and. . .reflect upon related instructional design activities” (Sugar, 2014, p. 48). 

Initially, 1-2 paragraph text-based cases were added to various sections of the course. These were used in 
classes to prompt learners to identify performance problems. Then, multimedia-rich cases were added to stimulate 
data analysis thinking. Learners completed these cases outside of class. The instructor conducted a debrief session in 
class. 

The presenters recognized that these static cases were helpful. However, learners were unable to connect the 
dispersed activities to authentic ID practice. A new low-tech, face-to-face, interactive, and progressive case was 
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designed and implemented in an early class session to supplement the text-based and multimedia homework cases. 
Based on instructor reflections and evaluative data, the progressive case was successful in setting the context of ID 
practice, fully engaging learners in content, and supporting their connecting theory to practice. 

Interactive and Progressive ID Case 


Purpose 

The progressive case was intended to build upon what learners experienced in the static cases, not to 
replace them. “Case-based instruction holds pedagogical promise for the education of instructional designers, who 
must be prepared to exercise a high level of problem-solving skills, critical thinking, and reasoned judgment in their 
work” (Ertmer & Russell, 1995). The progressive case requires more problem-solving (identifying a variety of 
possible problems and solutions), critical thinking (questioning statements of stakeholders, choosing ID activities), 
and reasoned judgment (adjusting assumptions, opinions, and recommendations based on emerging information) 
while progressing through an ID project. 

“Case-based instruction tends to involve complex problems, situated in the real world of practice” (Ertmer 
& Russell, 1995, p. 24) and requires “novices to practice ‘thinking like’ professionals” (p. 25). Thus, the interactive 
and progressive case was designed to be authentic (including interactions and problems from real cases) and ill- 
structured (requiring assumptions when information is missing, accepting multiple answers). 

Structure 

The case consists of a Powerpoint presentation with characters represented by speaking avatars (Voki, free 
online tool that has text-to-speech functionality). The Powerpoin presentation has a minimalistic design, including 
only purposeful design elements. A small icon on the top corner of each slide indicates progress within (and 
connection to) the ID process. 

Delivery 

An ID doctoral student facilitated the progressive case during class. The professor observed, providing 
additional feedback. The presenter introduced the case, facilitated the presentation, elicited learner responses, and 
led the debrief session. The learners were highly active, taking more class time than was allocated to the case. The 
case took approximately 2.5 hours of class time; 1-1.5 hours was intended. 

Scenario 

The case involves a performance problem that occurred in a corporate setting: Customer service 
representatives (CSR) were transferring callers with product problems to the wrong technicians, or dropping calls. 
The manager (Voki) presented the problem and asked the instructional designers to provide CSRs with training on 
questioning skills. Learners were asked to describe what they thought of the manager’s request and their next steps. 
During further analysis, learners received information about the company and previous training. Speaking Voki’s 
simulated mini-interviews with employees. Throughout, learners were prompted to respond and reflect on the 
meaning of new information (e.g. What is/are the problem(s) and cause(s)? How do you know?). As intended, 
learners provided new answers as they learned more about the scenario. The case continued in similar structure 
through project phases. The case ended with a reflective debrief prompting learners to think about the entire process 
and how it will relate to the larger class project they were about to begin. Throughout the rest of the course, 
references to the case were made during activity debriefs. 

Lessons Learned 

Learners completed surveys on levels of teacher, social, and cognitive presence and a short formative 
evaluation. The following tends emerged from these data and instructor/designer reflections: 

a) Learners were highly engaged, providing multiple and varied responses. 

b) Learners enjoyed the interaction and case resources/media. 

c) Learners developed a conceptual understanding of the nature of the ID practices, and the importance of 

asking questions and seeking information. 

d) The interactive nature of case prompted a lot of learner engagement (asking unanticipated questions) 

leading to focused discussions that were longer than planned, indicating high social/cognitivc presence. 

The following design enhancements are being considered: shorten case to allow for a longer debrief; provide 
more information about case context. 
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Conclusion 


This instruction provided learners with an authentic, interactive, and progressive case that required them to 
think like ID professionals early in their training. This practice proposal provides support for unique low-tech and 
low-cost instruction, requiring only Powerpoint and free online software that are rather easy to merge into a 
progressive case. This type of case -based instruction could be used in a variety of contexts and content areas. 
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What room is there for human agency in a world increasingly infused with Big Data, predictive analytics, 
and responsive technologies? In theorizing about e-learning, what emphasis is placed on peoples’ abilities to act 
beyond the pushes and pulls of theory-posited factors and variables? This paper addresses those questions, 
introducing human agency as a construct and then exploring its role in understanding practice in four contexts of 
inquiry: practical reasoning; change and innovation; learning and instruction; and human values and social justice. 

In each context we briefly show how agency erupts and disrupts either established theorizing or prevailing notions 
of a technology-driven world. As a complement to both learning theory and technological determinism, attention to 
agency can carve out space for surprise and the unexpected - essential for true learning and change to happen. 

Learning technologies have grown so rapidly in recent years, we are still looking for appropriate 
interpretive frames to make sense of things (Anderson, 2008; Siemens, 2005). Our purpose in this paper is to revisit 
foundational thinking in educational technology, highlighting the role of human agency. An agentic perspective 
presents a fresh way of seeing our work, with the potential of re-invigorating the project of informing and improving 
e-learning thinking and practice (Yanchar & Spackman, 2012). 

Human Agency Defined 

For our purposes, human agency refers to: people ’s capacity to act and shape their worlds in a given time 
and place, in pursuit of individual and collective purposes. Agency is concerned with our ongoing engagement in 
the world - with the material world and with each other, individually and collectively, as we pursue various human 
goals and activities. In some respects the construct of agency reflects some longstanding questions about free will 
and the nature and consequence of human choice and action, brought to the forefront by Enlightenment thinking 
about reason, causality, and God’s receding prominence (see Yanchar, 2011). More recently agency has proven 
useful in the social sciences, particularly in theories of activity and practice (Emirbayer & Mische, 1998; Kaptelinin 
& Nardi, 2006). 

A central concern involves the extent to which our actions are determined by various internal and external 
factors. Following an agentic perspective, a significant part of human engagement in the world is not reducible to 
smaller or larger-scale structures, theories, or causal processes, real or hypothesized. Practice divided by theory 
always leaves a remainder. Brain research or cultural studies or organizational theory or educational principles - 
these may help us understand a particular situation, but the people themselves and the always highly specific details 
of the situation also play essential roles. An agentic perspective is not dismissive of attempts at scientific 
explanation, but rather seeks to complement those patterned explanations with equal time and respect for the choices 
and actions of people as they encounter the particulars of a time and place. 

Agency and E-Learning 

In educational literature, agency often refers to individual students empowered as active learners within 
learning ecosystems (e.g., Holden et al., 2014). At other times agency is used to denote principled activism based on 
a moral stance to achieve social change (Campbell, Schwier, and Kenny, 2005; Schwier, 2004). These framings are 
useful, and we explore them further below. At a deeper level, however, we see people as fundamentally active 
agents in all settings and conditions. Agency is not only a special case for effecting social change or engaging in a 
particular kind of learning environment; agency is how people engage the world productively, even when they do 
not feel particularly active or moral or in control of our environment. In spite of constraints, agency plays a central 
role in human activity, to learning in general, and to e-learning in particular. 
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Our tech-driven stories 


Like every field of practice, e-learning depends on some key stories to explain and justify its existence. 
These meta-stories generally draw on the power and promise of technology as a tool for education and learning: 
E-learning as reform. Education is broken - too many students failing, too many 
unprepared or incompetent teachers, too much boring seat-time, too inequitable, and too 
expensive. And not relevant enough to the real world. E-learning is the fix. 

Education as a cutthroat business. Schools, like countries, must compete for attention, 
for prestige, and for business. Schools that do not invest in technology will become less 
attractive and less relevant over time - and will eventually go out of business. We have 
no choice - we must invest in e-learning to stay competitive. 

Innovation drives the future. Technology is fast becoming so pervasive and powerful - it 
will determine our future more than anything we can do ourselves. E-learning is how 
technology is manifest in education - hence e-learning is our future. The sooner we 
engage and commit, the better adapted to the future we will be. 

Each of these stories is compelling and familiar. Each story is also “true” in many ways: education does 
have increasingly publicized challenges, university administration is under incredible external pressures, and we are 
living in highly technologically innovative times. Within this context e-leaming certainly does offer some potential 
solutions. Yet each story also paints a very particular version of reality with real consequences for how we think 
about the rights, roles, and responsibilities of individuals. Technology is indeed powerful, but we need to also ask 
where people fit within these narratives. Seeing people as active agents engaged in co-constructing their worlds 
serves as a useful complement to the potentially homogenizing narrative of technological determinism. 

Practical reasoning in the world 

Ken Wilber (2001) presents a simple way of framing human endeavor in terms of scale (see Table 1 

below). 



Inside-Out 

Outside-In 


I 

It 

Individual 

Psychological 

Spiritual 

Physiological 

Behavioral 


We 

Its 

Collective 

Relational 

Structural 


Cultural 

Social 


Table 1. Individual and collective perspectives, adapted from Wilber (2001). 

Viewed from the inside out, we tend to see relations in inter-subjective terms: “you” and “I” construct 
meaning and “we” engage in activities through shared understanding. The shift to the outside-in perspective adopts a 
more “objective” stance. A social network, “it,” consists of various structural and process variables from which “its” 
meaning is developed. 

Human agency comes from a place where the structures and processes of human relations are given and 
played out - but even with those structures in place, people find surprising ways to connect and act together. The 
relations are personal and experiential, and the way we relate is not just logical but interpersonal and meaningful and 
often illogical from outside frames. We are more than objects in a model or even characters in a story - we have a 
tacit understanding of situations that transcends propositional knowledge. 

This close-in view has implications for learning and knowledge construction. Knowledge is created and 
shared wherever people engage actively and purposefully: in ETRD articles, but also in UGA faculty meetings, in 
Google’s halls, and in AECT annual meetings. It happens in the blogosphere, on Twitter, and the IT Forum listserv. 
And especially, knowledge creation happens in classrooms — online, face-to-face, and hybrid; seminars and 
MOOCs, museums, workplace cubicles, and brown-bag meetings all over the world. 

Reasoning itself is affected by a close-in view of collective activity. Abstract, quantitative, and reductive 
forms of reasoning have been privileged since the Enlightenment. An agentic perspective assigns equal value to 
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applied forms of reasoning in everyday affairs: running households, mediating conflicts, brokering a business deal, 
managing a classroom, scouring the Internet for information to guide a decision. This practical thinking (Aristotle’s 
phronesis or practical wisdom) uses a variety of explicit strategies (e.g., costs and benefits, problem-solving 
routines) and intuition or tacit knowledge (cf. Polanyi, 1958) - along with a healthy dose of emotion (Roth, 2009). 

Activity theory (cultural-historical activity theory or CHAT) posits that people act intentionally to pursue 
ends within complex systems. The careful analysis of everyday work and performance contexts is the main inquiry 
method used by CHAT researchers. Agency can be seen as an individual level or in groups. How individuals and 
groups interrelate and share agency is a subject discussed among CHAT researchers (Edwards, 2009; Matuso & 
Marjanovic, 2014; Yamazumi, 2009) and other practice theorists (e.g., Bourdieu, 1977; Schatzki, 2001). 

Change and innovation 

Change and innovation are essential terms for e-learning specialists, as noted in our meta-stories referenced 
above. The term “change agent” occurs more than 3 million times on the Web and refers to someone whose job is to 
facilitate change. Because e-learning is so tightly identified with emerging technologies, we are often asked to play 
the role of change agent in introducing e-learning products and services. Transformational change, the kind that 
redefines deep structures and processes, occurs less often but has become a serious object of study in business and 
technology. 

Some change models focus on information dissemination and people’s rational use of that information 
(e.g., Rogers, 2003). Other theories see change as cultural assimilation and adoption of tools and practices (Wilson, 
et al., 2001). Change theories can sometimes carry a pro-change bias, assuming that new technologies or innovations 
are inherently valuable. Critical perspectives, in contrast, can help us appraise the risks and possible unintended 
consequences of change proposals. Both ways of thinking about change can be productive - considering potential 
positive, negative, and neutral outcomes. E-learning specialists in a change-agent role, therefore, should be aware of 
these multiple perspectives and values, in order to better serve clients and the profession. 

Human agency becomes especially important for technology adoption when we realize that it is not just a 
neutral tool that gets adopted. Rather, technology change involves a whole set of values, practices, and perspectives 
necessary for the tool to be used productively in ways that make meaningful differences for people. These 
differences can be micro — an individual learner’s achievements; meso — an organization or group’s growth; or 
macro — positive social change at a national or global scale. 

Instructional design grew out of a tradition of active problem solving in the world. Our theories are 
prescriptive, we say; our plans and efforts are interventions. Both terms have a connection to medical diagnosis and 
treatment. The student or patient is lacking or ill, and we are here to diagnose the problem and determine an 
appropriate solution or fix. This generic problem-solving orientation is reflected in ADDIE; in our applied research 
models (e.g., needs assessment, performance gap analysis, formative evaluation; return on investment); and in our 
technology adoption and change models. 

Of course, we now know that complex systems resist top-down control and externally imposed change. 
Some systems theorists even avoid any use of problem-solving language, referring instead to the emergence or 
evolution of features and affordances. And some more flexible methods have emerged: rapid instructional design, 
agile project management, appreciative inquiry. The Web has accelerated the emergence emphasis, to a point where 
innovation and change are often associated not with top-down designs but with crowd-sourced development and 
sharing of open educational resources (OER). 

Learning and instruction 

Human agency has deep roots in learning theory. Working within an information-processing tradition, a 
generation of learning theorists explored ways that people pursue, monitor, and regulate their own learning (Ryan & 
Deci, 2000; Schunk & Zimmerman, 2008). As a capstone to an impressive career in this area, Albert Bandura 
explored some learning implications of human agency (Bandura, 2001). At the same time, educational 
anthropologists saw learning both as a shift in how people see themselves and their abilities to participate in 
community (e.g., Lave and Wenger, 1991). These two theoretical traditions have contributed to a number of 
instructional approaches that seek to empower individual students (and often teachers) as they engage in self- 
directed and problem-solving activities. 

One current manifestation is the maker movement, typically framed as a means of training students for 
STEM professions. An ongoing initiative at Harvard (Agency by Design, 2015) frames maker learning in terms of 
three elements: 
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Capacity - students have the skills and knowledge needed to shape their world through 
design activities 

Inclination - students have the disposition to engage in self-directed and collaborative 
maker activities that lead to tangible and virtual products 

Sensitivity - students notice occasions and opportunities in their worlds where maker 
activity is appropriate 

Knowing and being able to design useful and cool things in a studio or lab is not enough: students also need the 
disposition or frame of mind, and then linking to their everyday worlds by noticing and acting on opportunities for 
engagement. The authors speak of these three elements together as maker empowerment (Agency by Design, 23015, 
p. 5). 

Active pedagogies like makerspaces, constructivist environments, and problem-based learning put human 
agency front and center. Many of the current trends in technology-enhanced learning emphasize self-directed 
learning: personalized learning paths based on individual preferences; universal design for learning which is 
sensitive to individualized approaches to the same learning outcome; competency-based learning which removes 
arbitrary specifications for seat-time and grants more control to learners and their use of individual abilities. While 
other methods such as direct instruction or lecture- or text-based learning are not so obviously agentic by design, 
learners can still exercise control within these more constrained environments. Indeed agency may be more 
important in these constrained environments, because learners must actively make connections to received content to 
make it worthwhile (Morse, 2010). Every learning situation involves pushes and pulls, opportunities to act and to be 
“acted upon,” submission and mastery. For students to be fully prepared and resourceful, they need coping strategies 
to respond productively when they are not fully in control. Again we emphasize: human agency is not a special case, 
reserved for the right pedagogy or opportunity. Agency describes how people are, how we engage the world, and 
how we learn. 

Equity and social values 

Educational studies have largely turned away from value-neutral, scientific investigations of strategies and 
methods, toward seeing education as a response to structural inequities in our institutions and society. Critical 
pedagogies and critical theory in general have become mainstream ways of seeing educational problems. From this 
lens, education seeks to empower all students to resist cultural systems that curtail their full participation. Students 
exercise their agency by claiming their voice and actively responding to their worlds. 

Issues of social justice and equity permeate e-learning theory and practice, sometimes referred to as the 
Digital Divide. When students have an option to use technology, some will and some will not or cannot. When 
someone develops a curriculum or chooses a tool, some will be advantaged and others left out. Casting our gaze and 
remaining fully open to the full range of effects of technological innovations - this is the responsibility of e-learning 
specialists as professionals and leaders. 

Physical access is only part of the challenge: students also need access to cultural capital and ongoing 
support for their development. Specific support and guidance with technology can be especially empowering to 
disadvantaged learners. Glynda Hull reports on the use of digital storytelling to give students a chance to tell their 
stories, thereby seeing themselves as active achievers able to shape their worlds (Hull & Katz, 2006; see also Davis 
& Weinshenker, 2012). 

Just as people can be seen as engaged or passive - so can theories or models of education. Theories can 
reflect agentic or inclusive values in varying ways and degrees, and we should be conscious of those differences. 
Similarly with our educational technology foundations: learning theory, traditionally proffered by white male 
theorists, has enjoyed a near hegemony in framing core issues of learning and instruction. Traditional learning 
theories, however, are facing challenge from critical theorists of feminist and critical-race persuasions (Sandlin, 
2005). This overall turn to greater access and participation in the conversation about learning is consistent with the 
goals of social justice and inclusion. The point here is not that technological changes and innovations or learning 
theories are themselves inherently oppressive but that we must think critically about ends and impacts, perhaps 
especially the unintended consequences. An agentic perspective is comfortable in this cross-border terrain, and helps 
legitimize the infusion of values into professional practice - empowering students to use learning as a means of 
gaining full participation; and encouraging designers to think critically and reflectively about their work. 
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Agency extended 

We have defined agency as a human quality, but agency can also be constructed as a quality of non-human 
objects and artifacts. Bruno Latour is a French thinker whose actor-network theory places humans as agents within 
complex networks of human and non-human agents (Latour, 2005). Human artifacts in particular exercise a form of 
agency; they “have a way of exceeding or changing the designs of their makers” (Shafer & Clinton, 2005; see also 
Tenner, 1997 and Postman, 1993). Think of our world in terms of actors and agents and pushes and pulls. My boss 
wants me to. . . My students are asking me. . . This Web browser wants me to. . . This LMS tries to get me to. . . 
Material and cultural artifacts are designed for a purpose; everything has properties or “agendas” that constrain and 
afford action. We live in a world where we must assert our own agency amidst the various efforts to act upon us. 
This is the topic for another paper, but the idea should be enough to invite reflection about e-leaming practices, and 
how designers, instructor-facilitators, and learners can all make good use of their agency in applied settings. 

Concluding Thoughts 

Among philosophers, free will and agency have long been subjects for theoretical debate (Yanchar, 
Spackman, & Faulconer, 2013). We would argue that the issue remains relevant for practice. A student who feels 
empowered to act freely and assert his or her will upon a situation - that student will likely do better in class or on a 
test (Dweck, 2006). Likewise teachers, learning designers, and educational change agents with an agentic 
perspective will likely be able to function effectively within challenging environments with distractions, constraints, 
and completing pushes and pulls. Thus we believe both theoretical and practical advantages can come from agentic 
thinking as it concerns ed-tech and e-learning practices. 
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1. Active Learning and Faculty Development 

At present in Japan, emerging assertion to change qualitatively university teaching exists. Peoples are interested 
in introducing the active learning together with traditional methods in teaching, and also in faculty development. 

According to the ERIC thesaurus scope note, active learning is "Learning in which the learner is the principal 
driving force, with the instructor (if one is present) as facilitator of the process -- among the many active learning 
approaches are experiential learning, cooperative learning, problem-solving exercises, writing tasks, speaking activities, 
class discussion, case-study methods, simulations, role playing, peer teaching, fieldwork, independent study, library 
assignments, computer-assisted instruction, and homework". The documents published by Japanese Ministry of 
Education also lists methods for active learning like problem solving, group discussion, fieldwork and so on. The list in 
those statements show approaches through which active learning may take place. 

Also, according to the ERIC scope note, faculty development is “Activities to encourage and enhance faculty 
professional growth”. The documents published by Japanese Ministry of Education states faculty development indicates 
organizational effort in which teacher improve contents and methods of her/his lessons. It is necessary to examine the 
effect of faculty development on the improvement of lessons. 

Referring to concepts of active learning and faculty development, the author tries to search issues to improve 
subjects and methods in teacher education by examining the particular approaches for active learning by the author. 

2. Kinds of Teaching Methods Used 

The author worked for lessons of teacher preparation education like “freshman seminar”, “method of education”, 
“educational evaluation” and so on. 

The method he most frequently used is the explanation using copies of textbooks. He also presented recordings 
of lessons of elementary/secondary education. Also he tried to use approaches for active learning described in the 
followings. 

In this report the author discusses instances in the freshman seminar class. The freshman seminar classes are 
called in Japanese as “Manabi no Gihou I” and “Manabi no Gihou II”. Each class is perfomed in one semester. In those 
classes, reading activity is mainly used. Each students read essay about education, wrote resume and made presentation 
to colleagues. In the “Manabi no Gihou I”, teacher selected articles and each student are assigned the article she/he reads. 
In the “Manabi no Gihou II”, each student selected the article. Student chairman managed the presentation and the 
discussion follwed. 

Those activities are thought to correspond to group discussion, cooperative learning, writing tasks, speaking 
activities and library assignments. 

Teacher partly explained about how to write the report, how to refer to published works, the way to write 
bibliographic data of the works and so on using copies of several textbooks. 

In addition to those methods, teacher tried to introduce puppet play and English conversation using Skype with 
cooperator living abroad. 

3. The Media as the Method of Active Learning 

There are many kinds in media. In the Cone of Experience by Dale, E., “Dramatized Experience” has its 
position. The puppet play may be used for dramatized experience. In the present activity, the teacher assumed that 
introducing puppet play is useful for students to develop “Educational Tact”. 

In the “Manabi no Gihou I” class of 2013 spring semester, computer terminal room is used as the classroom. 
Computer terminal room is thought to be a kind of media. Teacher could send examples to students and each student 
could send report to teacher. 

Computer software as the Power Point and the Skype are also kinds of media. Using the Power Point, each 
student could design materials for presentation. And use of Skype made it possible students to communicate with person 
who live distant site. 

Those media could be used in approaches for active learning. 
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4. Lesson Contents and Methods are Derived from Curriculum 

College curriculum is constituted to fulfil diploma policies. Each lesson is a part of the curriculum. 

The freshman seminar classes, “Manabi no Gihou I” and “Manabi no Gihou II” are introduced in 2013 academic 
year. The theme of “Manabi no Gihou I” are “To read and to talk” plus “to investigate”. The theme of “Manabi no 
Gihou II” are “to write and to inform” plus “to investigate”. 

It is highly professional work to write the syllabus of the lesson, and therefore it is a subject in professional 

growth. 

5. Improvement through Faculty Development 

The author attended many faculty development activities through his carrier as university teacher. At his present 
position, the activities of FD were the lecture and the group discussion in 2013 and the panel discussion about relation 
between acquirement of research fund and FD in 1014. The present study is triggered by the FD in 2014. Questionnaire 
for class assessment is provided. However at present the author could not use the result sufficiently. Those are subjects in 
professional growth. 

6. The Approaches for Active Learning Depends on Teacher’s Feature 

Active learning that could be introduced were those originating from teacher’s feature. Since he has had contact 
to a circle of puppet play for a long time, he could make the stage of puppet play and get information about pieces of 
puppet play. Also because he had interested in the field of CAI and had activities for fairly long time, he could easily use 
current CAI tools. That is to say, the active learning a teacher can introduce depend on the teacher’s character. 

7. Motivating Students 

The difference in student participation were rather large in the approaches for active learning. Motivating 
students to those activities is important. It is a subject in professional growth. 

8. Priority Consideration 

We could find several subjects in using approaches for active learning as described above. Those could be the 
theme of FD and also the subjects for teacher himself. It is almost impossible to make full provisions to those all. Priority 
and balance considerations about those subjects are necessary. 
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